Introducing RTFM (Real-Time Frame Model): a highly efficient World Model that generates video frames in real time as you interact with it, powered by a single H100 GPU. RTFM renders persistent and 3D consistent worlds, both real and imaginary. Try our demo of RTFM today!

Generative World Models will inevitably be computationally demanding, potentially scaling beyond even the requirements of today’s LLMs. But we believe they are a crucial research direction to explore in the future of rendering and spatial intelligence.

RTFM does not build an explicit 3D representation of the world. Instead, it takes one or more 2D images as input, and directly generates new 2D images of the same scene from different points of view.

RTFM can be seen as a learned renderer: it is an autoregressive diffusion transformer trained end-to-end on large-scale video data, and it learns to model 3D geometry, reflections, shadows and more just by observing them in its training set.

RTFM can also be used to reconstruct real-world locations from sparsely captured photographs. These are not real videos: they are frames generated by RTFM.

For a limited time, you can try out a live demo of RTFM yourself, hosted on cloud GPUs and streamed to your device (mobile support included!):

Lol, you only need $40,000 of GPU to do it.

> generates 2D images of that scene from new viewpoints without building any explicit 3D representation of the world > RTFM circumvents this problem by modeling each frame as having a pose (position and orientation) in 3D space 🙃

RTFM. 🤭

Amazing to see this live!

reality will have a very different meaning in a short time

@pbaylies this literally came out while we were talking about it.

Woah ✨

Impressive.

Wow that’s cool !

Love where this is going—want to brainstorm together?☄️💭

Pretty good

Is everything always in focus or can you alter focal length and aperture settings as you can with a camera?

What for?

>real time >15fps anyone whos used a vr headset that runs <60fps will know this'll make 99% of people vomit within half a minute of use

First clip felt like Subnautica 😊

Impressive work! Real-time 3D world modeling used to rely on multi-GPU clusters. Now architecture and system design are catching up with model innovation. Maybe the next breakthroughs in AI will come not from larger models but from smarter systems that scale efficiently.@scitix

It'll become obsolete once Genie 3 is out (assuming DeepMind will ever release it)

But why do it this way and intentionally not store a representation of 3D geometry?

RTFM beach!

it's not 3D

With only a single H100 is massively impressive 💯

I remember when they did this with Minecraft, but the world reset if you looked at a block or turned around.

Very impressive

Is this a shared experience? If not, then when? If so, I'd like to see demo video of two people - located in different parts of the real world - inside this virtual AI space (show their POV side-by-side). After that, I'd like to know how it scales with increased participants.

I think you don't know the meaning of efficiency

Surely I can't be the only one who remembers that RTFM used to mean something else?

Generated from 1 photo?

Mind-blowing demo!🤩

RTFM performs horribly on RAG benchmarks...

This is insane.

nice

this is nuts wow

Is the Universe rendering itself at speed of light?