I'm blown away. This AI filmmaking workflow for precise camera control, multiple characters, and dialogue is insane: 1. Generate a start frame in Midjourney 2. Match the poses in Blender, animate the camera 3. Feed both to Seedance I didn't think this would work. Two consistent characters, solid performances,... the move tracked perfectly through the entire beat. Even the soup looks great.show more

What I never understood about these is why you would waste so much time in Blender when you can just napkin sketch this on a piece of paper and take a pic

For simple stuff you're totally right, but when you need precision control it's amazing that you can get there manually. I'm super comfortable in blender so it wasn't a huge time investment.

Instead of matching the pose, could you not take the image and generate a depth map, then just project it into 3d and animate around that? Seems like that would be a less effort workflow

Interesting, I'll have to try that. What software would you use to make the depth map from the image?

No idea but presumably midjourney could do it? I don't use any of them so I don't know, but seems like something they could make a good attempt of. I assume it doesn't need any real accuracy, and is mainly so that when animating the camera you have some vague idea of what it's pointing at.

Ah nah Midjourney can't generate depth maps - MJ is excellent for aesthetics but not currently very adept at processing or editing imagery. But I'm looking online and there are some other tools that can do it! I'll experiment.

Or you could like... get a camera and shoot it for real. This is a simple shot in a kitchen. There's no need to fake it.

This shot, yeah for sure. But this is just a simple test as I'm just experimenting with the pipeline - there are many obvious instances you can imagine where it wouldn't be so easy to film

Not really. Real filmmakers make real films. Start small, work your way up.

No need to gatekeep my man haha. Are you a filmmaker? I’ve worked in film for 15 years, and I produced this feature film that premiered at Tribeca and distributed to theaters nationwide

Have you ever tried the reverse: prompting to turn a photorealistic video into a 3d render so that you can better map image adjustments onto it without running into the problem of how using realistic video as input can often introduce low res noise or unwanted detail?

I have not! That's an interesting idea haha. I actually haven't yet tried using realistic video footage as the reference material. I assume the low res noise and stuff will get solved as the models improve.

maybe i should finally pick up blender

Playing with 3d guided AI since late 2024 (controlnet was a pain, Seedance it's like cheat mode). It's great.

too much "empty air" at screen right. Your frame is off and should be panned slightly left from the start. The boy eating soup is the subject of the shot , and he ends up very on (left) edge at the mid point of the shot.

Fair enough! That's a critique of my camera animation though, not the workflow. This was just a quick test for experimentation :)

Yes you are definitely on the right path... unlike the fellow who sugessted storyboards on napkins. In Industry we use Autodesk Maya for camera blocking but Blender is perfectly fine.

sick I've been trying to crack that, did you also use Claude to create the Blender scene or did you do it by hand ?

By hand! The scene is just two mannequins, roughly positioned to match the start frame, and a plane for the table.

Check your dm :)

This technique is really cool because of the amount of control it gives you over the characters. Great lesson here.

The 3D/AI hybrid workflow will likely be the future meta. I imagine that a lot more 3D work will be involved for large productions though. They still need way more control than what we are seeing here.

100%. I have a background in 3D animation, very comfortable in the traditional sphere. I want as much control as possible. Bright days ahead!

Good luck with the second shor where all the furniture are misplaced and the face slighty change enough to be uncanny

I actually generate start frames for every new shot by hand, one by one, so there's no drift :)

How many takes did it take you to get this shot?

I think I generated 4 takes, just making little tweaks to the performances. Camera was perfect from the first generation

That’s awesome. Would you be willing to share your prompt structure?

Loving these samples for Seedance 2.0. How long did it take for you to animate the scene in Blender?

This one maybe like 30 mins?

This demo gets clean camera moves and consistent characters by posing the whole shot in 3D first, then handing it to the model just to render. The impressive part is old animation skill, not the AI. tell me I'm wrong 👀

This looks awful😬

the future

As a former layout lead it would be a good idea for you to render your Blender camera with a HUD displaying film aspect ratio and the camera cross hairs at the center of the frame. I can see a looseness in your move that would get a kick back from a supervisor.

Interesting flow. Blender seems to be a great middleman for a lot of AI video.

This is amazing. Have you found that the set stays pretty consistent from scene to scene?

I don't use multi-shot features from the video models - I make start frames for every shot, one by one. The image models like Nano Banana Pro and GPT-2.0 are great at spatial consistency. Sometimes I'll do a long shot with Seedance where the camera moves around the space and I pull start frames from that.

cc @DW_MidJourney

lmao the Seedance sequence doesn't even follow the camera movement of blender

i need to try it

Why Blender, why not any video you’ve made?

Doesn't have to be Blender! But working in 3D means fine tune control in a way that would be difficult to film.

what platform do you use for Seedance ? Midjourney?

I currently use Magnific for Seedance generations. Midjourney doesn't have an API so you need to use their own platform for that.

Wow, thats a great idea. Nice work.

Impressive pipeline, thoughtful execution.

Thanks Shara!

Amazing work. How elaborate was your prompt for Seedance? And did you use it through Runway or another platform?

I use @magnific instead of Runway. Here's my exact prompt for this one :) Use @ img1 as the clean start frame. Don't change anything about the frame. Use @ vid1 as a camera motion reference. Follow the camera movement in the reference exactly, but add some handheld camera shake. You can move the characters. The woman looks at the hunched over man with concern. The man picks up the spoon with his left hand and slowly eats the soup. After a moment, the woman says "You doing ok?" The man says in a low voice "I'm fine." The woman says hesitantly "We should talk about it." And the man looks up at her and says "Why are you acting like you care?" No music.

@magnific thanks for sharing

This is fantastic 😍

Thank you!!

You're welcome 😁

blender previs is the secret. most people skip this step and wonder why image-to-video drifts — the model needs a solid anchor

This is why AI filmmaking won’t just replace production tools. It’ll wrap around them. The best results will come from people who know when to prompt, when to constrain, and when to hand control back to the model.

I'd love someone doing the reverse. Take the best scenes from the best movies. Digitalize them. Then we have a digital library to make a new movie

hideous. the reason people love movies is not because they "look just like real movies". when that clicks, you'll be on the road to something.

Honestly the camera move staying locked that clean the whole way is what got me

I need more detailed instructions than this

1. Learn the basics of blender. How to move around the viewport, how to add cubes and objects and move them around. How to add a camera, and use keyframes to animate it over time. Tons of YouTube tutorials on this stuff 2. Use Midjourney to create an image you’re happy with as the first frame in your shot. 3. Back in blender, roughly place cubes and stuff to match that start frame. Or do it the other way around, but the former is easier. Animate your shot, and then export a “viewport playblast” so you don’t need to render 4. Upload the start frame and the viewport reference to seedance and prompt what action you want to happen

That’s all well and good I don’t think I would ever pay to watch this, or recommend it to anyone beyond the novelty