This - in my eyes - is the future of rendering: A coding model (LLM) creates a rough outline of the game with simple geometry but without shading. Then a video AI model generates the final beautiful image. Meet "Code World Model":

I agree. I Experimented with this a bit. In a few years the AI accelerator hardware will support this with better quality than deterministic rendering.

Very cool!

Interesting, but how do you handle latency and consistency between frames when a video model owns the final pixel pass? Seems like a hard constraint for interactive rendering.

Temporal consistency can be achieved by considering previous frames.

That is nice and if you have a site you built for that: - FREE to list and if you want to or see a competitor listed, advertise on their listing page.

I don't see the point of the coding model here. You can take a game engine, make it generate the proxy part, with full designer control, and then render it with the model. Am I missing something obvious? Also, I'm not sure how different it is from Lucy model from decart.

The coding model makes it possible to prompt the scene

Also , coherency is still an unsolved issue, as far as I know.

Well, that's exactly what this technique aims to solve.

It solves that were a character should be, it will always be. But the appearance of the character is still subject to changes. You can spot it in the videos, the background scenery is not consistent. Don't get me wrong, it's interesting. But there is still a lot of work !

@mark_k I think you just need a backwards pass where you put a diffusion generated results back into the code to store it

@mark_k Maybe I understood wrong, but you can't store all the frames of your game. The point is that if a player is going to a location he already visited before, but everything has changed, it's just a bad experience. A model like genie3 only have 1 minute buffer.

@mark_k You modify your original geometry with vastly more detail based on the generation. This means you don't need to store per frame data, everything is already in the 3D model. That's the point

@mark_k Interesting idea. It means you need a runtime video to 3d model. In this case, isn't it better to use a world model that generates geometry in the first place, like marble?

Possibly. But the physics and object persistence. Ideally there is a data layer of context behind anything that is generative that is authoritative. Where the rules live. You could potentially make the output just do that magically but somewhere you have to know that a persistent thing exists and you want it to keep existing.

That's why it's hybrid

I’m making something that might kind of be like that. Eventually. But I think it might be more baked than inferenced. Like you generate stuff and it becomes the context base and the inferencing is the last layer. Like a reshade. But you populate enough base that if you don’t have the reshade it still works bare bones. And the AI layer is really the still the lighting and particle effects. So the thing that nvidia is doing with the faces now. When it does that. A bunch of the face bakes in. Just a thought. It’s all an optimization problem in the end. So we shall see how that happens. Balancing these things to the performance of common denominator hardware. I think the adaptive authoring will come before the full inferencing. But who knows. Maybe m not sure the required size of the models to be generalize reshaders. Fun to think about.

Wooowiee!! 😍

nice idea probably correct visuals deterministically derived from code doesn't capture the 'art' while Image Generation does

Nice. can you elaborate the tools used?

Check the quoted post

Thank you for highlighting. 🙏🏼

The progress is crazy. Less than a year ago when the genie model came out there was no world consistency. When you turned around the scene would completely change and that looks almost solved now

@mark_k this is wild! could change how we think about game design. real-time adjustments might get easier.

La visión de la que te llevo hablando 2 años @regniblod

how would real-time interactions and input fit into this pipeline?

You can see the interaction in the video. It's input for the coding model.

This is super interesting. so how does the whole pipeline work? If a video model creates it - how do we bring it back into the engine to make it interactable? Or is that not required at all?

That's why it's hybrid

I haven't really had a chance to understand how these workflows actually pan out. This is really cool - will look it up. Thanks!

this is very interesting

The first stage could be a JEPA model, this should be way more efficient

The interesting part is how fast the baseline for “impressive” is moving. What looked like a portfolio piece six months ago now feels like a prompt experiment.

models will converge eventually

Code World Models separate structural generation from visual rendering, enabling cheaper iteration.

This workflow is really good but so is just recording yourself or even legos doing stuff and then converting that to the final

Thanks, Mark! I actually believe that even simple geometry may no longer be necessary in the future. Instead, the video model could be conditioned on bounding boxes paired with text prompts, all automatically generated and refined by the coding agent.

Right on target, its super cool

of course this is where its going. Going one step further - you could just use AI to generate all the objects/assets on the wireframe map as 3d models. This would ensure a consistent look for everyone if necessary. Like - dream a little bigger.

This is super cool. I've been working on something similar. Would love to chat if you have the time!