Introducing The Matrix --- a foundation world model for generating infinite-length, hyper-realistic videos with real-time, frame-level control: - Infinite-length video generation - 720p high-quality rendering - Real-time, frame-level control at 16 FPS - Generalization to real-world video control 🔗Blog: 📄Paper: 💻Code & Playable Demo: Coming soon! Key Innovation: A... brand new technique called the shift-window denoise process model, enabling auto-regressive generation for diffusion and consistency models in real-time. Special thanks to project leader Ruili Feng and the entire Matrix team for their dedication and hard work over the year-long project.show more

We compare The Matrix with many state-of-the-art game simulators.

Interestingly, pre-trained on a vast collection of internet videos combined with AAA game footage, The Matrix demonstrates impressive domain generalization. For instance, it enables scenarios like driving a BMW X3 through an office area.

Here’s an example showcasing The Matrix generating an ultra-long video with precise real-time control lasting over 14 minutes (>13440 frames). For more examples, visit our project page:

Amazing! Looking forward to the release! 🔥 If you do open model checkpoint release as well, then I’d be happy to help you with that from @huggingface side 🤗 My DMs are open! Let’s make this huge!

@huggingface Thank you for the offer. 🤗

in some sense it's a 1D generation (or fake 2D), same view won't show up twice when you look back at it again.

Everything about gaming, mobile gaming and everything else is about to change. Can't wait for the code drop people are going to go crazy with this :)

nice! however "consistency models in real-time" need to also apply to the backgrounds, the mountains completely changing in 5 seconds after the camera turns isn't going to work. Every asset needs to remain consistent - unless the goal is an acidtrip experience for the end user.🫠

is also at X, the first author of this project.

@Yuchenj_UW host on hyperbolic