Video yükleniyor...
Video Yüklenemedi
Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our... show more
32,350 görüntüleme • 1 gün önce •via X (Twitter)
38 Yorum
[1/7] 🧠 Why code? Video models keep getting better visually, but two gaps remain: physics and precise control. A thrown ball may hover, a pendulum drift off-period, or a collision look plausible but break physical laws. Prompts and ControlNet-style guidance still struggle to specify every actor’s pose, object’s size/color, scene layout, motion path, and camera timing. Executable programs make these constraints explicit and testable before generation.
[2/7] 🎬 Borrowing from film previs: a coding agent turns a prompt into an executable Three.js scene. Layout, object motion, camera trajectories, and event timing live in one editable program. The rendered proxy video provides the spatiotemporal plan; a reference image sets the target appearance. The video model adds geometry details, materials, and lighting. Code defines how the scene evolves.
[3/7] 💻 Code → previs → inspect → revise → video. The agent edits objects, parameters, and event logic, then checks the updated proxy before generating the final video. Here, redirecting the robot by changing route waypoints—while preserving the rest of the scene—is as simple as one line code change. Once the program is ready, proxy rendering takes under a minute, making shot design quick to validate and iterate.
[4/7] 🔍 The challenge: a useful proxy is visually crude. Feed it directly into a reference-to-video model and its low-poly geometry and simplified materials can dominate the output. We want its layout and motion, while letting the reference image and pretrained model supply the final appearance. How can conditioning separate the two?
[5/7] 📊 Our key observations with MiniMax-H3 frozen: • The pathways are complementary: Qwen-VL has a stronger effect on appearance; VAE contributes more to motion and structure. Dual-path works best overall. • Proxy conditioning matters most in roughly the first 25% of denoising. Early structural guidance leaves a lasting trace in Query features after the proxy is removed. • Swapping conditions reveals a clear division: appearance mainly follows the reference image; layout, viewpoint, and geometry mainly follow Proxy-VAE. Remove the proxy too early and structure slips; keep it too long and the low-poly look persists.
[6/7] ⚙️ Our solution: training-free, stage-wise conditioning. Keep the full proxy and both native conditioning pathways during early denoising to establish structure and motion. Then remove the proxy while text, the reference image, and model priors continue refining appearance. Early guidance persists without carrying the synthetic look through every step. The video model stays frozen; no extra control module is trained.
[7/7] 🚀 One code-driven workflow, many applications: Anime, bullet time, gaming, scene editing, robotics and trajectory control, architectural and product cinematography, physics-grounded motion, and 3D/4D reconstruction. From freezing the action while the camera keeps moving to editing an object’s route, it starts with an executable scene. More demos 👇
@raywzy1 Fantastic work! So happy that we share the same insight. Code will be the future prompt of visual generation.
@raywzy1 🛫
@jon_barron @raywzy1 Great direction! Ultimately we should detach intelligence from video generation.
@raywzy1 Great work! It really reminds me of LLM-grounded Video Diffusion Models that my team released ~3 years ago. Seems like an advanced version of it with code as the grounding proxy! Congrats!
@raywzy1 Well done!! Looking forward to testing this out.
@raywzy1 Congratulations @raywzy1 ! Amazing work!
@raywzy1 @grok wtf is this
@raywzy1 AI video is slowly turning into something developers can actually debug.
@raywzy1 Yeah pain point
@raywzy1 awesome work led by @raywzy1 !!!
@raywzy1 Yes yes this is what I have been looking for. Making a video where the outcome of future scenes are conditioned and controlled by prompts and code depending on the past scenes.
@raywzy1 Building the set once and planning each camera angle from it could make multi-shot scenes so much easier
@raywzy1 incredible!
@raywzy1 Putting layout, motion, timing and camera paths into editable code makes video generation something you can diff.
@raywzy1 Using code as the intermediate previs layer finally gives us actual deterministic control over physics and camera work.
@raywzy1 Genius idea !!!
@raywzy1 Awesome.
@raywzy1 Great work! Coding is the right way to capture physical dofs. How much time does it take to render the scenes? Do you think it’s deployable in real-time robot control?
@raywzy1 Agentic coding as a controllable video engine code for physics and camera diffusion for appearance brilliant abstraction
@raywzy1 This fits agents better than one-shot video generation. When scene logic and motion stay in code, iteration and debugging remain practical.
@raywzy1 Incredible breakthrough turning code into controllable video is next level!
@raywzy1 This is incredible work code as the control layer is genius
@raywzy1 well done, I would like to test it
@raywzy1 Code as engine for video control is brilliantly nerdy innovation
@raywzy1 cool idea. curious how you'd code a smirk on that chef's face
@raywzy1 The mix of code and video opens up so many options.
@raywzy1 Making video generation programmable opens up an entirely new level of creative control
@raywzy1 can we later just remove the diffusion layer completely ?
@raywzy1 agentic coding as engine for controllable video generation is perfect
@raywzy1 Combining programmable control with video generation opens incredible creative possibilities
@raywzy1 awesome work !
