Meet SceneSmith: An agentic system that generates entire simulation-ready environments from a single text prompt. VLM agents collaborate to build scenes with dozens of objects per room, articulated furniture, and full physics properties. We believe environment generation is no longer the bottleneck for scalable robot training and evaluation in... simulation. Website: 👇🧵(1/8)show more

A hierarchy of VLM agents builds scenes stage by stage: Layout → Furniture → Wall-Mounted Objects -> Ceiling-Mounted Objects → Small Manipulable Objects Each stage is implemented as a collaboration among 3 VLM agents: Designer, Critic, and Orchestrator. User study with 205 participants → 92% average realism and 91% average prompt faithfulness win rates against baselines! 🧵(2/8)

The object vocabulary is unbounded. SceneSmith routes each request to the right strategy: - Static objects → text-to-3D generation - Articulated furniture (cabinets, drawers) → retrieval from an articulated object library - Materials → retrieval or generated Yoga studios, pet stores, hotel lobbies — any scene, any object, generated on demand. In one of our scenes, SceneSmith generated 798 objects in a single room! 🧵(3/8)

Every object is fully physically simulated. To prove it: earthquake test. Plates slide off tables, books fall from shelves, and glasses topple. Every object is metric scale, has collision geometry, and VLM-estimated physical properties (mass, center of mass, inertia, friction). <2% inter-object collisions. 96% of objects are stable under gravity. Baselines: 3-29% collisions, 8-61% stability. 🧵(4/8)

One of many applications: automatic robot policy evaluation across environments. 1. Task description → LLM generates diverse scene prompts 2. SceneSmith builds all environments 3. Robot executes policy in each scene 4. Evaluator agent verifies success using sim state + vision No hand-crafted success predicates. 99.7% agreement with human labels. 🧵(5/8)

Here's the evaluation pipeline running. We compare a policy against a degraded variant to show that we can differentiate between the two policies. We tested across 100 generated scenes spanning four different pick-and-place tasks. We hope that such systems will help bring more rigorous evaluation to robotics. 🧵(6/8)

SceneSmith exports to any major robotics simulator (MJX, USD, SDFormat). Here is a Rainbow RBY1 being teleoperated in our scenes. Opening cabinets, grasping mugs, navigating rooms. Third-person view (left) + robot head camera (right). 🧵(7/8)

This is an amazing collaboration with @cohnthomas43, @ZakharovSergeyN, @RickCory21, @RussTedrake 📄 💻 🔧 Explore our interactive 3D scenes on our website or download them from Hugging Face! 🧵(8/8)

Really cool work! We had a very similar work Architect which also focuses on indoor scene generation (also specifically for small &amp; manipulatable objects placement): Different method applied to similar problems, excited to see this new progress!

Very cool! Thanks for sharing. Using image priors is a promising way to improve spatial reasoning.

Add some steam VR teleop I would give you some free teleop data^^

Love that!

This looks great! Nice videos also. Would be cool to hook up a motion planner in here with a task sampler to generate diverse demonstrations for VLA training

Agreed. Maybe even some VLM-aided TAMP or similar to get diverse data from such a model-based planner. @cohnthomas43 set up a simple version of a model-based planner for these scenes, but we have only used it for evaluation so far.

release quality is amazing! congrats!

this is incredible! Out of curiosity, have you used Scene Smith to generate RL environments in which to train robots? D’you have any demos of how one could train/ post train for robotics using SceneSmith?

We have not tried RL-based training yet. We do have teleop demos on the website that could be used for supervised learning. RL would be exciting to try!

Thanks for sharing! I'll check out the website. Another question: how d'you benchmark or evaluate your model?

this looks incredible, congrats Nicholas!!

Thank you! It ended up working way better than what I had hoped when we last talked about this

One thing I’m especially curious about is how far this kind of unbounded, VLM-driven object vocabulary and physics estimation can go before you start needing task-specific, human-curated distributions again—for example, for household manipulation vs. warehouse vs. surgery—and whether we’ll see “benchmark overfitting” in simulation the same way we did in vision and NLP

Agreed. I think we are at the point where we have to push the robot side to see how far we can get with these environments and whether they start breaking down anywhere.

Earthquake test was my favourite part in this thread! 😆 Cool stuff!

Glad you liked that one! A pain to simulate with those large forces 😂

Procedural generation used to be a massive bottleneck. This looks like a serious shortcut for dev teams.

So cool!

@Scobleizer Holy fk… I thought embodied AI won’t be real for another 20 years. Looks like robots might be in my house just after 10 years

Very cool... Although did anyone else notice that the knife was backwards on the table? 😁 There will always be a fundamental lack of actual understanding in ML/AI.

Agreed here. VLMs seem to struggle with spatial imagination. Setting a table with place settings that face in different directions (and not just toward the current image render) is a revealing case of this. Image generative models are much better at this. Maybe the next version will use agentic video models?

i just want to redecorate my kitchen with this

hello hi yes i need this

Great work! May I know how to teleoperate a mobile robot in simulation (with only one person)? Is there any mature pipeline for this?

You should be able to send commands from your real-world teleop system to the simulator instead of to the robot directly. We did open-source a lightweight mobile iiwa teleop example with a space mouse a while back: However, there are much better recent approaches that use a VR headset for this teleop. Also, check out this company:

@GChongkai how much does training in simulation translate in real world performance. ?

@GChongkai Recent works have shown that this is very promising. For example: I do think that you will want to co-train on both sim and real data jointly to get the biggest benefit from added simulation data.

@GChongkai I also saw a post from somebody to create a gaussian splat of your personal space which can be used for simulation. This would be better suited if you want to train in a specific environment vs what you have here if you want to build something general purpose, right ?

Correct. We worked on real2sim (replicating an actual environment in simulation) in the past: The goal of this approach is to match the distribution of real-world and simulated environments for increased scale, but it might not be great at replicating one particular environment from images (though we have used it for that as well).

Very cool! How long does it take to generate each simulation-ready scene if you generate one scene per run?

I haven't ever timed this 😅 However, we implemented a bunch of performance improvements targeting throughput over latency. Hence, it wouldn't be much faster than when generating ~25 scenes or so in parallel. The biggest bottleneck is API response times. Hence, we have an option to opt into OpenAI's priority tier that speeds this up by 50% but is twice as expensive. Switching to Gemini Flash (or other speed-optimized models should also make a big difference here).

vlm agents scaling to dozens of physics-ready objects per room is a clean unlock for sim workflows. but joint params and collision fidelity drift fast in multi-gen cycles without persistent validation layers. 🇳🇴

Great work! Curious about how do you make the physical params of simulated objects plausible? And how do you evaluate?