Our research introduces a system that enables you to generate 3D environments from text prompts and train embodied AI agents within them! Website: Code: How did we leverage Objaverse assets to create interactive 3D environments? 👇

To enhance object retrieval and placement accuracy, we ask OpenAI's GPT-4-Vision API to annotate the following attributes of Objaverse assets: 1/

Our layout design module places objects in stages: (a) floor objects, (b) wall objects, (c) small objects, (d) ceiling objects 2/

To place floor and wall objects, we sample a scene graph with LLMs where each edge represents a spatial constraint between two assets. The task of arranging assets is then a constraint satisfaction problem (CSP), which can be optimized using mathematical solvers. 3/

Similarly, we leverage the common sense priors of LLMs to suggest placements for small objects. This is followed by a surface object placement function, which enables randomized and efficient positioning of smaller items onto receptacle objects. 4/

To import Objaverse assets into AI2-THOR (@allen_ai ), a Unity-based embodied AI framework, we developed a pipeline comprising: (a) mesh reduction, (b) visibility point generation, (c) collision mesh generation, (d) texture compression. 5/

Check out our embodied AI experiments in our paper! Joint work with the wonderful team: @YueYangAI *, @LucaWeihs *, Eli VanderBilt, Alvaro Herrasti, @WinsonHan , @nickhaber , @jiajunwu_cs , @RanjayKrishna , @LingjieLiu1 , Chris Callison-Burch, @yatskar , @anikembhavi , and Christopher Clark. Paper: 6/

Dear fellow scholars 🗣️🗣️🔊🔊🔥🔥

One holiday break away from someone wiring up a roguelike with this idea

How hard would it be to constrain the exterior shape of the environment? If this can generate floor plans within a pre-defined envelope I have immediate customers 👀

I would love to learn about the specific use case you have in mind!