Introducing Universal Simulator (UniSim), an interactive simulator of the real world. Interactive website: Paper:

UniSim is trained on broad data rich in different axes (internet text-image, human activity videos, simulated executions, real robot videos, panorama scans, etc).

UniSim supports both high-level language actions and low-level motor controls through conditional video generation.

UniSim supports simulation of diverse manipulation actions such as various actions in cooking a meal.

UniSim also supports diverse navigation actions.

UniSim supports diverse environment transitions such as different objects being uncovered.

UniSim supports training RL agents purely in simulation that transfer to real robots in zero-shot.

UniSim also supports training embodied VLM planners purely in simulation and transfer to real robots in zero-shot.

Work done with @du_yilun, @coolboi95, @JonathanTompson, Dale Schuurmans, @pabbeel.

@coolboi95 @JonathanTompson @pabbeel Learning realistic world models is becoming a reality @ylecun

UniSim also supports long-horizon interactions without compromising simulation quality.