Testing LCM LORAs in an AnimateDiff & multi-controlnet workflow in ComfyUI. I was able to process this entire Black Pink music video as a single .mp4 input. The LCM lets me render at 6 steps (vs 20+) on my 4090 and uses up only 10.5 GB of VRAM. Here's... a breakdown 🧵[1/11]show more

Entire thing took 81 minutes to render 2,467 frames, so about 2 seconds per frame. This isn't including the time to extract the img sequence from video and gen the ControlNet maps. Used Zoe Depth and Canny ControlNets in SD 1.5 at 910 x 512. [2/11]

Improving the output to give it a stronger style, more details & feel less rotoscope-ish, will require adjusting individual shots. But doing the entire video in one go lays down a rough draft for you to iterate on—build on fun surprises, troubleshoot problem areas. [3/11]

For the input video I used every other frame in order to target 12 fps. [4/11]

Here's a screen shot of how I added the LCM LORA. I went with the baked in VAE from the checkpoint. [5/11]

Kept the prompt pretty generic to see how it would apply to all the various shots. [6/11]

In the K Sampler, I used the LCM Sampler. You need to update to the latest version of ComfyUI to access it. [7/11]

And here's how I arranged the nodes for multi-control net. [8/11]

If you want to learn more about LCM LORAs, I mainly referred to @NerdyRodent’s tutorial. Go check it out! It speeds up all rendering in SD. It's not just for videos! [9/11]

If you want to learn more about Animate Diff, go check @PurzBeats’ live stream videos! [10/11]

Lastly, shout out to @rainisto for giving me the idea to try this on a full music video, and @PurzBeats again for answering some of my questions about AnimateDiff! [11/11]