Introducing Large Video Planner (LVP-14B) — a robot foundation model that actually generalizes. LVP is built on video gen, not VLA. As my final work at Massachusetts Institute of Technology (MIT), LVP has all its eval tasks proposed by third parties as a maximum stress test, but it excels!🤗

Given an observation and a task description, LVP generates a video plan of a human hand or robot gripper accomplishing the task. We then extract the actions to deploy on physical robots. The video provides a strong prior for the action. (1/5)

Unfortunately, all the best video models are built for content creation instead of robotics planning, not to mention open-sourced ones. So I decided to build an open foundation model for the robotics community to explore this direction. (2/5)

LVP combines Diffusion Forcing and History Guidance to achieve strong consistency, instruction following, and generalization. Even with limited data and GPUs, we get crazy generalization as video models absorb all the human activity knowledge online. (3/5)

As the final work of my PhD, I want to make it convincing. So instead of cherry-picking a few tasks myself to make a cool video, I let third-party testers pick whatever task they like, be hard, be creative. LVP excelled in every test, proving true task-level generalization.

I’d like to thank my co-authors @tianyuanzhang99 @HaoranGeng2. Without your help, such a big project would be impossible. This is my final gift to academia, and I hope you will love it. Visit our website for details and try our HF demo 🤗 at

@MIT this is a huge breakthrough

@MIT such an amazing effort on the open source video model done right! q: if you ask an instruction like grab a cup from the cabinet, the video model may hallucinate a cup on the shelf after opening the cabinet if there is no cup there. how would u plan to address this?

@MIT Such an impressive work. Congrats!

@MIT Impressive work with LVP-14B! Exciting progress in video gen. Congratulations!

@MIT This feels like a quiet but important shift, video as the substrate for robot intelligence just makes sense

video gen for robotics > VLA is the right call. video models understand physics and temporal dynamics implicitly from training data. LVP using video as planning representation is brilliant - you're basically using the world model learned during video generation for robot policy. this is the future

@MIT Impressive leap! How does it handle scenarios with different robots?

@MIT Impressive work showcasing true generalization skills!

@MIT Pretty sick!

@MIT Impressive work on LVP-14B! Truly groundbreaking.

@MIT LVP generalizing video generation to video planning is the breakthrough. Most models are VLA-bound. Building on video gen, not VLA, is the right architecture for real-world robot tasks. This approach scales better across diverse environments. MIT + robotics = paradigm shift.

@MIT Impressive work! Excited to see the impact of LVP-14B in the field.

@MIT Impressive work on LVP-14B! Excited to see how it performs in the real world.

Hi Boyuan, LVP is a strong result. Video as the primary modality for robot foundation models makes sense, and the zero-shot generalization to novel scenes is impressive. The question I keep coming back to: you trained on internet-scale video of human activities. What happens when the target domain has zero internet video?

@MIT Excited work! Pre-training video foundation model for robotics have more implicit physical prior and better zero-shot capability. I strongly believe this is a promising way to break down the sim2real gap and training reliable robot policy.

@MIT Great work. LVP-14B's generalization on third-party tasks is a significant advance.

@MIT That's super cool man

@MIT sounds really cool, congrats on your work at mit