1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger... model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSourceshow more

2/5 📊 Data. Data is the key to success🔑 We built a large-scale, standardized data pipeline that curates, filters, and dedups broad public robotics and egocentric human datasets. This gives us a clean training dataset with diverse tasks, scenarios, actions, and embodiments.

3/5 🦾 Method. We leverage a 2D kinematic skeleton rendering as a unified control signal. From robot arms to human hands ✋, every embodiment is rendered as an image-aligned 2D skeleton. This enables high-precision action conditioning for video models. 🎯

4/5 📈 Results. With our cleaned data and kinematic skeleton conditioning, we finetune the Cosmos2.5-2B model on a single GH200 GPU ⚡. Our method delivers superior quality on action following, appearance, and motion consistency, pushing SOTA performance with a fraction of the compute. 🏆

5/5 🧪 Robot policy evaluation. Finally, we deploy OSCAR for real-world robot policy evaluation using the episodes from RoboArena. Our extensive experiments show a high correlation ✅ between virtual assessment in our world model and real-world results, paving the way for a future where robot policies are evaluated entirely within generated environments!!

@wuzy2115 Impressive, it seems much more efficient. Is that because it introduces no new modules?

@wuzy2115 No new modules is one part of the efficiency. The most useful things are: (i) curating a balanced and diverse dataset, (ii) skeleton conditioning as a unified action representation, and (iii) the base model is only a 2B model (Cosmos 2.5)

@wuzy2115 Have you looked into multi-camera generation?

@wuzy2115 We haven't looked into it yet, but it's a great future direction!

@wuzy2115 Love the name

@wuzy2115 the skeleton is an interesting representation, i wonder if there are any ambiguity problems when the three red arrows are colinear (their plane intersects camera center)?

@wuzy2115 Good question! Yes, this will indeed cause the ambiguity; one potential way of fixing it can be to use three planes to indicate the gipper (e.g., the triplane idea, and each plane has three red arrows).

@wuzy2115 a world model is the agent rehearsing what happens if i do x before committing. video prediction is just what that rehearsal looks like from outside

The 2D skeleton-as-control choice is the clever part — it forces the conditioning channel to be embodiment-shape-invariant, so the dynamics head cannot smuggle embodiment cues through the action stream. Curious how that holds up under DoF mismatches, e.g. a parallel gripper and a five-finger hand sharing the same skeleton abstraction.

@wuzy2115 I understand that you guys trained the model with inputs of the 2d skeleton and the first frame to generate video outputs. My question is that during inference when you have to do a robot action for a certain input frame, you need the 2D skeleton right! How do you accurately get that ??

@wuzy2115 Great work! Thanks for using our dataset, AIRoA MoMa dataset @airoa_org !

@wuzy2115 The combination of cross-embodiment generalization and full reproducibility stands out here. It's encouraging to see strong robotics results paired with open data, open weights, and a compute budget that remains accessible to the broader research community.