Introducing DreamFAR, a world model trained from scratch using only robot data, without internet pretraining. It supports: 🎮 Real-time teleoperation ⏱️ Minutes of long-horizon rollout 🧩 Generalization across 1,000+ tasks 🧵 (1/8)

Try the online interactive demo! 🧵 (2/8)

Humans learn physics by interacting with the world, not by watching it. Inspired by this, DreamFAR starts from active experience: dynamics learned from scratch on robot-native data instead of passive internet videos. We present a contact-based memory mechanism and an architecture that enables live & long-horizon prediction that generalizes across diverse tasks. 🧵 (3/8)

DreamFAR models the physics across 1000+ contact-rich manipulation scenarios: 🔧 tool-use, 🤝 handover, 🧱 Lego assembly, 👕 T-shirt folding, ✏️ writing ... 🧵 (4/8)

Over an one-minute long autoregressive rollout, the prior works blur, hallucinate or drift within seconds, while DreamFAR persists the scene layout, the manipulated objects. In this example episode, it even predicts the writings on the whiteboard that is visually similar to the GT. 🧵 (5/8)

Evaluating policy on real-robot is slow and hard to reset. We show that DreamFAR can serve as a fast and reproducible proxy for policy evaluation. A policy can be deployed in the world model and perform the tasks for minutes. DreamFAR ranks different policies mostly as real world does, though being optimistic in absolute terms. 🧵 (6/8)

Since DreamFAR runs in real-time from teleoperation input, an operator can teleoperate inside the world model and record demos without a real robot. We finetune a π₀ policy on three datasets: 75 real demos, 75 simulated demos, or their union. Adding the simulated demos to real ones nearly doubles the policy performance on real robot. 🧵 (7/8)

Our memory mechanism is based on the keyframes selected at contact-making or breaking moments, when the robot-object interaction changes the scene structure. We ablate the memory mechanism on a task where objects constantly leave and re-enter the view. In the example episode, the robot drops the bottles in a bin, looks away and then looks back. Without memory, the bin is rendered empty; with sparse fixed memory the bin contains bottles but the wrong ones. DreamFAR remembers the appearance of all bottles and the rendering matches the real observation. 🧵 (8/8)

Work done during internship at Amazon FAR w/ @BotaoUMD @antoniloq @andrewhowens @JitendraMalikCV @SOTA_kke @ruoshi_liu

Special shout out to my wonderful mentor @ruoshi_liu for so much inspiration on the high-level idea and detailed discussions on the implementation!

Lessgo! Really cool work!

NEAT

@PhongStormVN I like the name. Really cool 🔥

The contact-based memory ablation with the bottles makes the point instantly. What GPU is behind the 10+ FPS live demo?

Really cool work! So impressive!

an interactive browser rollout demo that stays coherent on day one

With pose-based retrieval, what happens when several objects pass through the same spot? I'm curious whether memory can pull back an older object state after repeated rearrangements, beyond the bottle leaving and re-entering view.

teleoperating inside the model to record demos is the part I'd watch, demo collection is usually the bottleneck, not the policy

@grok 这篇文章里讲了啥，可以用来解决啥问题。使用了什么数据集。

Pure robot-data world model is wild. Curious how it handles distribution shift over minutes-long rollouts — does it drift or stay on task? Would love to see the eval code. @shuizhuyu

very cool, congrats!

Super cool! Congrats @_YimingDou on this🤩

Cool work! Congrats!