This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are... still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)show more

It's using ACE , Kokora , Qwen image , Qwen edit , Wan...

I have already a lot of idea that will make huge improvement to the global composition. I will share that next week !

input : Christina Rossetti – Remember (1862, excerpt) Remember me when I am gone away, Gone far away into the silent land; When you can no more hold me by the hand, Nor I half turn to go yet turning stay. style : watercolor sketch

input : TITLE :Alice’s Adventures in Wonderland; AUTHOR :Lewis Carroll. “Alice opened it, and found in it a very small cake, on which the words ‘EAT ME’ were beautifully marked. She ate a little bit, and said anxiously to herself, ‘Which way? Which way?’” .STYLE : realistic movie

and just to finish, there is not any retouch in any of this example at any steps; I wanted to show what AI can do by itself. but its where human is important, and to make it perfect, it would be possible and pretty easy to control each steps of generation before the final clip

input : ✦Treasure Island (Robert Louis Stevenson, 1883) “Fifteen men on the dead man’s chest— Yo-ho-ho, and a bottle of rum! Drink and the devil had done for the rest— Yo-ho-ho, and a bottle of rum!” style : realistic

@ComfyUI Wouldn't it be less work if you would do it manually instead of using Comfy?

@ComfyUI jesus that workflow LOL

@ComfyUI You need to make 1 tts for the whole thing not 1 per scene. The tts will sound much more natural. Get word timing from elevelabs response or use whisper x. This will elevate it a lot. Individual tts make the narration choppy.

@ComfyUI me explaining to my landlord why i need 5000kw to make shitty AI stuff instead of learning how to make good art

@ComfyUI damn wow

@ComfyUI End product looks like shit!!

@ComfyUI I'm doing a similar thing just with code and I'm not sure how you are doing it but I found I got a much better quality story if the llm wrote the full story first then break it into pieces/scenes etc in a secondary prompt. I use word count to roughly control the length.

@ComfyUI That is some next level noodling!

@ComfyUI Inspiring!

@little_bluefish @ComfyUI Incredible 🔥🔥🔥

@ComfyUI 😍

@ComfyUI Sounds incredible.

@ComfyUI Very impressive!

@ComfyUI That's really cool! With my 3090 I bet it would take quite a long time to get a final output. What card do you have (or cards) ?

@ComfyUI Would this sort of workflow ever be possible on a standard gaming level PC? 3000 series Nvidia GPU

@ComfyUI Wow this is awesome! Great work!

@ComfyUI Perfect video for TikTok ai slop

@ComfyUI Wow

@ComfyUI Excellent ! Very smart ! Can we discuss by dm ?

@ComfyUI cant believe this website is free

@ComfyUI No one cares, share the workflow or stfu

@ComfyUI When you accidentally send a text to the group chat that was meant for your crush. 😩

@ComfyUI This is so frickin cool man! Ps, I really enjoyed the voice turning into “2 order of the orange chicken” guys voice in the middle there 😂🤭💜