Skip navigation
I'm blown away. This AI filmmaking workflow for precise camera control, multiple characters, and…
0:00 / 0:36

I'm blown away. This AI filmmaking workflow for precise camera control, multiple characters, and…

Reid Hannaford

@reidhannaford3.4K followers

Visit on X
3,323 likesDownloadOriginal post

228,286 views

I'm blown away. This AI filmmaking workflow for precise camera control, multiple characters, and dialogue is insane: 1. Generate a start frame in Midjourney 2. Match the poses in Blender, animate the camera 3. Feed both to Seedance I didn't think this would work. Two consistent characters, solid performances, the move tracked perfectly through the entire beat. Even the soup looks great.

61 Comments

Comments from the original post on X (Twitter)

  • Martin Erlic's profile picture

    @SeloSlav

    What I never understood about these is why you would waste so much time in Blender when you can just napkin sketch this on a piece of paper and take a pic

    14 likes

  • Reid Hannaford's profile picture

    @reidhannaford

    For simple stuff you're totally right, but when you need precision control it's amazing that you can get there manually. I'm super comfortable in blender so it wasn't a huge time investment.

    33 likes

  • Lincoln Margison - Game Development's profile picture

    @LincolnMargison

    Instead of matching the pose, could you not take the image and generate a depth map, then just project it into 3d and animate around that? Seems like that would be a less effort workflow

    10 likes

  • Reid Hannaford's profile picture

    @reidhannaford

    Interesting, I'll have to try that. What software would you use to make the depth map from the image?

    1 like

  • Lincoln Margison - Game Development's profile picture

    @LincolnMargison

    No idea but presumably midjourney could do it? I don't use any of them so I don't know, but seems like something they could make a good attempt of. I assume it doesn't need any real accuracy, and is mainly so that when animating the camera you have some vague idea of what it's pointing at.

  • Reid Hannaford's profile picture

    @reidhannaford

    Ah nah Midjourney can't generate depth maps - MJ is excellent for aesthetics but not currently very adept at processing or editing imagery. But I'm looking online and there are some other tools that can do it! I'll experiment.

    1 like

  • Alexander Fedderly 📚's profile picture

    @Alex_Fedderly

    Or you could like... get a camera and shoot it for real. This is a simple shot in a kitchen. There's no need to fake it.

    5 likes

  • Reid Hannaford's profile picture

    @reidhannaford

    This shot, yeah for sure. But this is just a simple test as I'm just experimenting with the pipeline - there are many obvious instances you can imagine where it wouldn't be so easy to film

  • Alexander Fedderly 📚's profile picture

    @Alex_Fedderly

    Not really. Real filmmakers make real films. Start small, work your way up.

  • Reid Hannaford's profile picture

    @reidhannaford

    No need to gatekeep my man haha. Are you a filmmaker? I’ve worked in film for 15 years, and I produced this feature film that premiered at Tribeca and distributed to theaters nationwide

  • Adam Lieberman's profile picture

    @AdamJLieberman

    Have you ever tried the reverse: prompting to turn a photorealistic video into a 3d render so that you can better map image adjustments onto it without running into the problem of how using realistic video as input can often introduce low res noise or unwanted detail?

    4 likes

  • Reid Hannaford's profile picture

    @reidhannaford

    I have not! That's an interesting idea haha. I actually haven't yet tried using realistic video footage as the reference material. I assume the low res noise and stuff will get solved as the models improve.

    2 likes

  • SamAlτcoin.eth 🌞's profile picture

    @SAMALTCOIN_ETH

    maybe i should finally pick up blender

    3 likes

  • Rodolfo Fanti 🪐🖌️'s profile picture

    @3DRudy

    Playing with 3d guided AI since late 2024 (controlnet was a pain, Seedance it's like cheat mode). It's great.

    3 likes

  • verdoyant's profile picture

    @verdoyant_music

    too much "empty air" at screen right. Your frame is off and should be panned slightly left from the start. The boy eating soup is the subject of the shot , and he ends up very on (left) edge at the mid point of the shot.

    3 likes

  • Reid Hannaford's profile picture

    @reidhannaford

    Fair enough! That's a critique of my camera animation though, not the workflow. This was just a quick test for experimentation :)

    4 likes

  • verdoyant's profile picture

    @verdoyant_music

    Yes you are definitely on the right path... unlike the fellow who sugessted storyboards on napkins. In Industry we use Autodesk Maya for camera blocking but Blender is perfectly fine.

    1 like

  • Yann's profile picture

    @Yannlce

    sick I've been trying to crack that, did you also use Claude to create the Blender scene or did you do it by hand ?

    2 likes

  • Reid Hannaford's profile picture

    @reidhannaford

    By hand! The scene is just two mannequins, roughly positioned to match the start frame, and a plane for the table.

    3 likes

  • Yann's profile picture

    @Yannlce

    Check your dm :)