Generative AI is cool and all, but procedural 3D modeling just hits different. Check out this Houdini setup by Pepe Buendia. Why this is cool: Instead of manually placing every building and car, this system generates an NYC-style city that builds itself -- automatically spawning buildings with unique variations,... plus a traffic system where cars actually obey traffic lights and avoid crashing into each other (all orchestrated through houdini VEX code). Each car knows exactly which road it's on and makes real-time decisions about navigation. The whole thing runs on custom algorithms that handle everything from road directions to building placement. If you did this with generative AI, even approaches to conditioning and taming the chaos -- you'd still get a ton of variance which might be an issue for visual and spatial consistency. To me this is procedural modeling at its finest - precise control meets infinite scalability. Immensely useful for media & entertainment, but also synthetic training data for robotics and augmented reality. I of course can't wait until multimodal LLMs to have the spatial understanding to write these kind of controllable 3D worlds for us. Claude counting 2D pixels on a screen is a step in this direction. Soon they'll be able to manipulate 2D ortho viewports, and eventually spawn procedural 3D worlds.show more

If you liked this post, you might enjoy my writing on all things spatial intelligence and creative tech:

Since y’all are asking, here are my thoughts on a best of both worlds approach — the creativity and speed of generative ai w/ with fine grain control of a 3d scene graph

And here’s how you can do most of this in blender with geometry nodes

We should really be using machine learning as optimization steps where it makes sense to. Renderman's AI denoiser is a good example. Trying to build "world models" from whole cloth is just using the wrong tool for the job.

Agreed — why not use explicit 3d approaches as a foundation?

So cool. I don’t understand the finer details of procedural 3D modeling (still learning) but is the future with multimodal LLMs also that the rules and custom algorithms will be automatically generated? Where do you see this evolving? How much control needs to be explicitly defined by humans at the beginning?

yeah my hope is the LLMs read/write/edit a 3d scene graph with entity level control - so kinda like replit agent writing web apps for you - except with these trad 3d tools and engines. or you know the whole sora approach will pop out a world model and my preference is overkill :)

Even with this level of control this would largely be background midground work, most hero shots would have alot of manual work on top. For high end work, artists need control more than speed or ease of use.

No doubt — but Hollywood has inordinately high standards

You're coming around to the procedural side. Yay!