What if we can simulate an *interactive 3D world*, from a single image, in the wild, in real time? Introducing PointWorld-1B: a large pre-trained 3D world model that predicts env dynamics given RGB-D capture and robot actions. 🌐 from Stanford University NVIDIA

Actions are embodied, and deeply spatial of robot’s own geometry. To move the world, we don’t cast spells, click objects, or turn our heads. We move the hands we control. And we don’t think of our actions as flat numbers of joint angles or hand poses. We consider our “interaction geometry” with the environment, not in 2D but in 3D (e.g., consider reaching to hold a big box where hands are occluded). This also allows us to easily teleoperate robots with unseen kinematics or gripper shapes (with even tiniest differences). So, how should actions be represented for a world model? Our answer: *3D point flows*, flows that are sampled from known robot surface (from URDF) — we always know our bodies and the robots we built, so we can compute their geometries over time.

Physics lives in space and time, so should the world model. State is also represented as 3D point flows. Given RGB-D obs, we build a partial scene point cloud, concatenate it with temporally-stacked robot point flows, and feed it to a transformer (PTv3 @XiaoyangWu_ ). In one forward pass, it predicts per-scene-point trajectory for one second into the future. No object masks. No material priors. No photometric prediction. No trackers (for inference). Just a scalable backbone, for physics, for interactions. Trained with simple L2 loss but extremely dense supervision, across any embodiments, any tasks, any trajectories, for learning the single source of truth of the physical world. Akin to “next token prediction”, but not for 1D tokens, for interaction over space and time.

How can we get the data? Our observation: 3D vision is maturing for 3D world models. Thanks to advances in depth estimation (FoundationStereo @bowenwen_me ), camera pose estimation (VGGT @jianyuan_wang ), point tracking (CoTracker3 @n_karaev ), we built an autonomous pipeline to label accurate 3D point flows from robot interactions recorded with stereo cameras, such as DROID ( Further augmented with sim data from BEHAVIOR ( it amounts to ~500 hours of filtered interactions with accurate 3D — spanning single-arm, bimanual, whole-body, mobile manipulation — which will be fully open-sourced.

With data at this scale, we can finally ask: how to make 3D world models work? Metric is simple and telling: L2 error on point trajectories. With a large eval set, error bars are *micrometer* level (several times thinner than a human hair). Precise. Comparable. Progressable. We distilled what worked into a roadmap: modernizing backbone -> tuning training objective -> leveraging image features -> scaling up the model. And we see a clear scaling law: more data or larger model, better generalization to unseen rollouts. There is still a long way to go, but the signal is strong.

With such “human-hair” statistical precision, we then went on to study its generalization properties. We carefully analyze in-domain, cross-domain, and held-out domain (held-out lab from DROID), in both zero-shot and fine-tuned settings. Key takeaways: - For cross-domain, pre-trained PointWorld can be efficiently finetuned for both real-to-sim and sim-to-real transfer (20x fewer updates). - For held-out real-world domain, PointWorld zero-shot generalizes, surpasses specialists if finetuned. - Real-world data is irreplaceable, but real-sim co-training also benefits.

There are also more properties that I found intriguing. - Trained without uncertainty labels, the model emerges to capture action-conditioned uncertainty arising from object’s physical properties (larger variability along the edge of the cloth). - 3D point flow representation of action enables positive transfer across very distinct robot embodiments (single arm, a bimanual humanoid). - It works with just one camera, better with more, and benefits from additional cameras added at test time. - Chunk prediction improves accuracy and enables real-time inference. There is more than I can unpack here, so please check out the paper if you’re interested!

So, how can this be used on a robot? There are many ways to use a dynamics model, from sim2real RL, exploration, to policy evaluation, and many more. Yet what excites me the most is they enable *achieving novel goals at test time* by imagination — like Doctor Strange, but for robots. Given a task defined as cost/reward over points (can be done with VLM we use PointWorld in a sampling-based planner on an in-the-wild robot for tasks like non-prehensile pushing, deformable and articulated object manipulation, and tool use. All using a pre-trained model. From a single fixed camera. No object reconstruction. No demonstrations. No finetuning.

“What I cannot create, I do not understand.” — One pinnacle goal of spatial intelligence is to re-create the unique, physical, interactive 3D world we live in, understand it, and use it to enable the next generation of robots. Although I’m thrilled about the initial findings in this work, PointWorld is far from perfect, and we want to be transparent for others to also take on the challenge. To this end, we drafted a full page of limitations that we want to call for research and will open-source our data, code, and pre-trained checkpoints.

This project has been in the making for 18 months, the longest project I’ve ever led. None of this would have been possible without the invaluable support from the team @yu_wei_chao @a__mousavian @liu_mingyu @fox_dieter17849 @KaichunMo @drfeifei and compute provided by @nvidia . I want to especially shout out to @drfeifei for continuous inspiration, encouragement, and in-depth discussions, and to @KaichunMo for countless nights of hacking and help with large-scale training. Website: Paper: Walkthrough Video: Code (coming soon):

@Stanford @nvidia Cool work @wenlong_huang ! Congrats on the release! Curious what you think the method will perform for multi-fingered hands with more occlusions (assuming lots of data for pretraining ofc)?

Thanks Ruoshi! That's a great question! In our observation, contact-rich dynamics and partial occlusion usually do not pose a challenge (e.g., there is an "basket-shaking" example on the website which has some non-trivial dynamics, under quite a bit occlusion). However, it may need to be re-trained / finetuned with a higher resolution if used for hands

@LawrenceZhu22 @Stanford @nvidia Super cool work @wenlong_huang! Congrats! Curious if you think using sim data instead of a potentially noisy labeling pipeline might help?

@LawrenceZhu22 @Stanford @nvidia Thanks Sumedh! PointWorld is trained on both real and sim data, and we did some careful analysis of their properties in downstream eval. TLDR: (1) real data is still irreplaceable, (2) real+sim co-training also helps both quantitatively and qualitatively

@Stanford @nvidia I like the background music 😉

@Stanford @nvidia Cool and called it that point clouds would be important for fine grained trajectories

@Stanford @nvidia super cool technique guys!

@Stanford @nvidia This connects directly to what NVIDIA is building on the compute side. When physics, perception, and action all live in the same loop, the gap between virtual and physical collapses.

@Stanford @nvidia Woow! This is soo cool! Congrats! 3D world Models are 🔥

@Stanford @nvidia This is a really clean formulation: representing both state and action as 3D point flows (sampled from known robot geometry) feels like the right inductive bias for interaction-centric world models, and it’s refreshing to see “no photometric prediction” as a deliberate choice.

@Stanford @nvidia from a single image, a universe blooms—magic in every pixel. wild to think what stories our code will let us step into next.

@Stanford @nvidia PointWorld-1B is incredible! Real-time 3D simulation from a single image opens so many possibilities for UI/UX innovation. Can't wait to test this out. 🌐

@Stanford @nvidia Impressive. How general is it to different types of material/structure?

@Stanford @nvidia Since there is no assumption on material/structure, it all depends on what's in the data and how much there is 🙃

@Stanford @nvidia 这都是我们的自己的救命钱，你爸爸妈妈就这样过河拆桥，你们于心和安？你在San Diego读书几年住在我家，我们没有亏待你，从来没有涨过一分钱寄宿费。你现在读博士，还在大公司上班，你有大把前程！

@Stanford @nvidia interesting

@Stanford @nvidia 文龙，我是苏文。做人和做学问是一样的。你爸爸妈妈借的钱一直没有还给我，我希望你能在其中发挥作用。我知道你爸爸把钱转给你了，你现在装作没这个回事。人在做天在看，一定会讨回公道！

@Stanford @nvidia Congrats

@Stanford @nvidia that is cool

@Stanford @nvidia this sounds like a game changer for both gaming and marketing. imagine using interactive 3D environments to showcase products in a whole new way. the potential for engagement and storytelling is huge. can't wait to see where this tech goes!

@Stanford @nvidia Congratulations! Exciting work across multiple emerging areas of AI and robotics.

@Stanford @nvidia NİCEEE <3

@Stanford @nvidia that sounds like a significant leap for ai rendering

@Stanford @nvidia ... we would use it to train policies :)