MiniMax H3 Reference Facial Insanity One Stop Swap Shop

I'm replacing the sampling setup, currently getting much nicer output with er_sde/beta combo. Even 4 steps, with the 4step turbo of course, looks much nicer than restart sampling. I'd suggest trying that, and only enabling a turbo, for low steps, otherwise no LoRA at all for high. The ref LoRA is really only good with the full bf16 model. VBVR LoRA is awesome but it can totally ruin video face swaps. Just leave everything off to start.

Add the attention node of your choice after the loaders, should be able to cut sampling time in half. At maximum resolution, 10 seconds should take around 2 minutes. Unless you're not me, in which case I have no idea how long it should take.

There are a few ways to run this:

1) Your faces + voice, with a t2v style prompt will give you a t2v style output with your face and voice.

2) Your face + voice, with a reference image and an i2v style prompt will give you an i2v output. Ish. Not quite the same, though you can force it pretty hard. Generally I prompt it as a suggestion (eg. this is a general composition guide to the overall structure of the first frame...etc.)

3) Your face + voice + a source video will give you a face swap. I cannot emphasize enough what a great job this model is doing with swaps. The examples I gave do not even come close. Give it a minimum of four HQ pictures of someone. Notes on recording a good voice clone sample are below and in the WF.

There are three ways to supply the reference video: you can either

1) pipe it straight in (I didn't make this switchable but you can figure that out)

2) have the SAM detector make a mask for face/hair/etc. that will be used to make a composite that inverts the detected area. This tends to help H3 find what you want. This is default, I think.

2) have the SAM detector make a mask for face/hair/etc. that will completely black out (or white out if you like) the detected area. This can sometime be helpful.

I think when I've settled on a model and sampler and have attention/optimization added I'll post a nicer, more polished version.

Here's whatever I wrote before, though all requisite notes and links are in the WF itself.:

I think I forgot to link the SAM model in the WF. Here it is: https://huggingface.co/Comfy-Org/sam3.1/tree/main/checkpoints. At some point when I wasn't looking, comfy changed things up a bit with regards to SAM, so this is a checkpoint, and needs to go into checkpoints folder (duh). In case you're used to putting SAMs into their own folder.

My frame count logic isn't quite lining up the way I thought it would, so be careful with that if it matters to you. You can always just unplug and enter a manual cap in the loader and elsewhere as needed.

This workflow does things to your face.

Show the model your face. Give it a nice long look at your face.

Then tell it where you want your face to be. Have you heard of 'text to video'? It's just like this, with one small difference. It uses your face.

Oh, your voice too. You must give it your voice. Or a voice. Something.

If you choose to do something like the 'image to video' stuff, supply a reference image. There is some crap in there that will mask out the face of the reference so the stupid idiot model doesn't get confused, the moron. Point is, it's the same as above, except you're helping it out a bit with the composition of the video, how it should look at the start, and all that jazz.

It's basically like instant character training. Just feed it good data.

I'm switching to 20+ step full model generation now that the swap is working, have to see if H3 can actually put out a good image... what a shame that would be, an amazing likeness but garbage video, that would be appropriately Sisyphean, that's how all these models seem to work.

V2 Info - It will post whenever CivitAI's terrible content filter decides to rate my example video. Of course it will be rated X. You know, bathing suits and all that.

-Ha, wtf did I tell you? Rated X - how predictable was that...

v2 has a straight video face swap module. It's messy, I think face swap should be its own thing.

Be that as it may, it's working. Video is still pretty crappy, but I've just been focused on getting the swap to work.

My example faces aren't super high resolution, working with HQ real photos gives MUCH better results, cannot post that here though.

Should be able to dial in some quality now that I can get a consistent result.

Also there is no upscaling or refining or anything yet. You have to be super careful with that anyway when you do swaps. It's VERY easy to ruin a likeness.

Also needs attention nodes and all that jazz.

Every section has copious notes, you should be able to figure it out. I believe in you.

For obvious reasons I cannot show you how insanely good a job this model has been doing with real faces. So for the examples I had Gemini supply me with a few angles of a unique face. Well, unique as defined by an idiot model. So if it's hard to tell if the likeness matches, it's because A) Civitina is not real and B) even if she was you don't really know her so how the hell would you know if the likeness matches? So you will just have to trust me. I've been swapping heads for years, before diffusion models, doing it with my bare hands damnit, then all the way back to A1111 with original ReActor, I've tried absolutely every swap technique, and all of them together, plus loads of manual compositing with tedious roto masking. I've never seen anything close to the likeness matching I'm getting with this model.

So just give it at least four HQ closeups of your face, you can add more and shift the comp ref down, adjust prompt, mutatis mutandis, all the rest, you know what to do. This isn't for babies. It's not a complicated setup at all. If you don't have custom nodes, swap them out for what you do have.

Notes on each section are in the WF.

<SUBJECT> speaks with the exact vocal timbre, pitch, and voice identity of <Audio 1>.

IF AND ONLY IF your sample is crappy/noisy, tack on cleanser promptage like this:

<SUBJECT> speaks with the exact vocal timbre, pitch, and voice identity of <Audio 1>, but with clean studio isolation and zero background noise.

This is the text I use for voice cloning. It is designed to capture all the phonemes you need. Have people read it with a natural cadence, no special effort, just read it as they would if they were alone. With this model you can try multiple readings, however, so you can try reading angry or sad, etc. Might work if you need it, but my single source audio works GREAT.

---------

When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors. These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow.

---------

It really should be CLEAN. A good recording.

It should be stereo, 48khz. A normal reading should end up at around 30 seconds.