🤖 How can we scale up humanoid robot learning? Introducing 🌟VLK🌟: generating large-scale synthetic data with paired egocentric observations, text, and full-body G1 kinematics for learning humanoid loco-manipulation. No teleoperation needed! Website:

1/9 Inspired by human motion synthesis works such as HIHI ( and CHOIS ( which synthesize scene-grounded human-object interactions, we asked: Can such synthesized interactions become supervision for physical humanoid robots?

2/9 The challenge: at deployment, the robot must act from egocentric vision without access to the full 3D scene. Inspired by GaussGym ( we render realistic egocentric observations with 3DGS and use the paired data to train a vision-based policy.

3/9 We reconstruct 3DGS scenes, synthesize G1 and object motions from scene-grounded sparse waypoints, and render egocentric views in Isaac Sim. This produces paired egocentric observations, text instructions, and full-body G1 kinematics. Here are Walk To examples.

4/9 Our data generation pipeline produces 48,000 paired trajectories in 13 hours using 48 GPUs, with no human intervention! The core data modes include Walk To, Turn Around, Pick/Put (Floor), Pick/Put (Surface). Here are dataset examples of Pick (Floor).

5/9 We train a VLK policy initialized from π0.5 on the synthetic vision-language-kinematics dataset. At deployment, the predicted kinematics are used as reference motion for SceneBot, a contact-aware tracker that converts them into robot actions.

6/9 We built a simulation evaluation pipeline: MuJoCo runs a contact-aware tracking policy (SceneBot), while Isaac Sim renders new egocentric observations. This allows us to test VLK conveniently before real-world deployment. Long-horizon tasks in simulation are presented below.

7/9 We deploy VLK on the physical G1 for navigation tasks, where the robot is guided by egocentric vision and language instructions.

8/9 More real-world deployment results for box-lifting tasks.

9/9 Great collaboration within the Amazon FAR team! Collaborators: @wangyenjen @eric_srchen @TakaraTruong @ Pei Xu Advisors: @pabbeel @rocky_duan @KoushilSreenath @akanazawa @carlo_sferrazza @GuanyaShi @ckarenliu For more results and method details, check out our website and paper! Website: Paper:

The "no teleoperation" part is the real unlock here, teleop is the cost wall on humanoid data and most pipelines quietly depend on it. Generating paired egocentric + text + full-body kinematics synthetically is a much more scalable path if the sim-to-real holds. Curious how the learned policies transfer off the synthetic distribution. Nice work.

🤖🤔

@Z1hanW Cool!

So cool!

Fascinating approach. Scalable synthetic data generation feels like an important piece of making humanoid learning more practical. Excited to read more about VLK!

quá tuyệt vời 👌