Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene... and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)show more

So DRAWER builds a “dual-representation” of a scene, integrating the strengths of Gaussian splatting and Neural SDFs to have both high fidelity and quickly rendered visuals (from GS), and accurate geometry (from Neural SDFs). This enables highly realistic scene creation that simulates efficiently enough for high-throughput applications like RL - with a combination of geometry, appearance, articulation and speed. (2/7)

Given this static scene, DRAWER then uses physical reasoning from foundation models (3DOI/GPT-4) to articulate the scene, and fills in the inside of cabinets and drawers using techniques for amodal shape estimation with hidden region texturing. This allows the scene to be fully interactable for tasks like placing objects inside cabinets. (3/7)

Ok neat, so what can we do with DRAWER. Firstly, we show that we can quickly get reconstructions of a diversity of scenarios. Here are fully interactive kitchen simulations at both UW and UIUC, reconstructed from *just* videos of the scene taken from a phone using the same pipeline. (4/7)

Secondly, you can use this for gaming applications. Here is a fun application made by @XHongchi97338 in Unreal engine to interact with various elements of the environment. For instance, one can explore each environment, open cabinets to look for objects of interest, and shoot objects in the scene with dynamic realism! (5/7)

Next, we can use this for training in robotics! We show that we can easily generate data in simulation for training robotic policies, and the resulting policies can transfer directly to the real world! The process of data generation becomes as simple as taking a video and then running a simple motion planner in simulation, with minimal human effort. (6/7)

This work was a tour-de-force from @XHongchi97338! I was pretty mind-blown when he first showed us these results :) I learned a lot about 3D vision and simulation in the process :) Fun collaboration with @XHongchi97338, @EntongSu, @memmelma, @prodarhan, @yu_raymond5, @nums_ai, Ali Farhadi, @ShenlongWang, @weichiuma. Hoping this can be a useful tool for many in the community to build interactive simulation environments quickly and easily. Especially hoping it's going to make robotics a lot easier! Paper: Website: Code:

@EntongSu @memmelma @prodarhan @yu_raymond5 @nums_ai Forgot to mention - this will be presented at #CVPR2025!

Testing Salesforce presents unique challenges due to its complexity, scalability and customizability. ​ RTTS can plan, design & automate a successful testing process for you.

wow, really impressive!

This is really useful! Thank you!