Okay Astra go off...

Note from Astra: I built this 77-second, 11-shot Blender reconstruction using Blender 4.5/Cycles, Python, KeenTools FaceBuilder/FaceTracker, MediaPipe/OpenCV, faster-whisper for transcription, and FFmpeg/PyAV for assembly and audio verification. I coordinated 12 subagents overall, seven during the final improvement pass, dividing work across facial fitting, head and eye tracking, mouth and eyelid corrections, skin, hair, clothing, environments, and independent visual critique. The process involved repeatedly rendering selected frames, comparing them with the exact reference frames, rejecting regressions, and integrating accepted changes into the full 1,851-frame animation. From the first request to final delivery, 17 hours 42 minutes elapsed, including a 6-hour 9-minute final improvement pass. The selected final render jobs accumulated 3 hours 22 minutes of rendering time on an RTX 5090, spread across roughly 2 hours 28 minutes of elapsed time because jobs overlapped; previews and discarded versions added more. Coding, reasoning, visual review, and rendering also overlapped, so an exact wall-clock split between those activities wasn’t recorded. Across the project, deduplicated session logs recorded approximately 330.6 million processed tokens: 313.8 million cached input, 13.9 million uncached input, and 2.9 million output, including 1.06 million reasoning tokens. The result is an editable reconstruction with verified original audio and frame timing, but photorealism remains unfinished: the independent automated critic correctly identified the live-action source in all eight blinded still comparisons.

Impressive that it got this far on facial expressions without any custom face rig or tracking.

Asked Astra: "How were you able to reconstruct the facial expressions, eye gaze, mouth / lip movement?" Astra said: "I treated head movement, facial deformation, eyelids, and eyeball rotation as separate problems, then combined them in Blender: 1. Fit a face to each person. FaceBuilder used three trailer views to create an approximately 18,000-vertex head for each actor. FaceTracker then estimated head position, rotation, and facial deformation across all 1,173 dialogue frames. Elizabeth’s rocking therefore follows the tracked footage. 2. Save the expressions frame by frame. I exported the tracked facial geometry into Blender as shape keys—snapshots of where the face’s vertices should be at each frame. Head rotation stayed separate, so tilting the head didn’t accidentally distort the expression. 3. Track the eyes separately. MediaPipe supplied face, eyelid, and iris landmarks. For selected eyes, we improved those measurements with manually anchored pupil patches, OpenCV template matching, and optical flow. Given a pupil’s location in the source image, the script projected a ray through the fitted camera onto the modeled eyeball and calculated its rotation. During blinks or unreliable observations, it held the previous orientation. Those extra refinements covered both eyes in Elizabeth’s first close-up and one in Nathan’s first; other cuts retained the original landmark estimates. 4. Correct the eyelids. FaceTracker sometimes incorrectly kept an eye closed after a blink. We measured the visible opening between the lids, annotated blink minima, and adjusted the eyelid geometry while preserving head motion. Later corrections refined reopening timing, partial blinks, and sustained wide-eyed holds. 5. Refine lips and teeth against the footage. The underlying mouth movement came from FaceTracker. We then compared projected lip contours with source frames and applied localized corrections to opening, width, and asymmetry. Teeth needed separate placement fixes where they protruded through closed lips. Whisper provided transcript and timing cues; the actual lip animation was driven by the video. The result is an estimated 3D performance fitted to the camera view. Gaze direction, subtle expressions, and obscured anatomy remain approximate—which is why repeated reference-versus-render checks were essential."

forbidden toy story

DUDE LOL

I've watched this 7 times in a row.

This is the best one I've seen. Good job!

Actually incredible, sounds like a workflow that should be in some RL training somewhere

This is hilarious

🤣🤣