Simulation drives robotics progress, but how do we close the reality gap? Introducing GaussGym: an open-source framework for learning locomotion from pixels with ultra-fast parallelized photorealistic rendering across >4,000 iPhone, GrandTour, ARKit, and Veo scenes! Thread 🧵

Most vision-based locomotion relies on depth/heightmaps—easy to simulate but lacking semantics. Learning from RGB pixels has been too slow (NeRFs) or too unrealistic (traditional renders). GaussGym solves this with 3D Gaussian Splatting as a drop-in renderer.

Gaussian Splatting gives us speed AND visual fidelity. GaussGym supports motion blur, camera jitter, and shutter speed simulation—all at ~100K steps/sec for humanoid tasks. That's 4,096 parallel environments on a single RTX 4090.

Zero-shot sim2real transfer with no fine-tuning. We trained a Unitree A1 to climb 17cm stairs from RGB alone—trained entirely in simulation (in just one stage), deployed directly to hardware.

We leverage MapAnything for pose extraction, NKSR for high-res meshes, and @nerfstudioteam gsplat for crisp, photoreal 3D Gaussian Splats. Warp handles fast heightfield computation for terrain rewards and curricula. All processing is automated—from images to trainable scenes.

GaussGym unlocks semantic tasks beyond locomotion. Purple policy (depth) walks straight into the penalty area. Green policy (RGB) recognizes and avoids it. A simple task demonstrating how RGB is necessary for even “easy” tasks.

Thanks to @justkerrding, @arthurallshire, @JonasFrey96, @rocky_duan, @carlo_sferrazza, @pabbeel! Excited to keep pushing vision-based robot learning forward. 🔗 Website (and demos): 📄 Paper: 💻 Code:

Congrats, Ale! Very impressive work.

looking great

Impressive. Curious how stable the splat based rendering is under aggressive camera motion any plans to benchmark that?

@Ulas_eng It's quite stable! look at our website for examples

Very cool. The motion blur simulation is a nice touch too

Impressive that you can achieve zero-shot sim2real transfer so effectively. Makes the potential for real-world applications exciting!

Really cool project congrats! How difficult do you think it would be to make the objects (like a coffee cup) gradable by the robot?

Thanks! Super-imposing a mesh on top of gaussians can be done relatively easily by extracting the image + depth from the simulator, then laying based on depth values. This would slow down rendering in IsaacGym but would be fairly fast in IsaacSim

use @OVRtheReality

Cool!

Super cool!

@sarayu_sapa map anything used

Looks great! I'm considering doing gaussian splats -> object meshes for my project but am concerned the output meshes will be too noisy/non-smooth etc. How well does the NKSR library work in your experience? And do you load the full scene mesh into IsaacGym?

Cool idea! I've found NKSR works quite well, but can sometimes produce unexpected holes in the resulting mesh. I wrote some code to fix this and other mesh-related issues: