Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

GameFactory Creating New Games with Generative Interactive Videos present GameFactory, a generalizable world model that learns from a small-scale dataset of Minecraft game videos. By leveraging the prior knowledge of a pretrained video diffusion model, it can create new games in an open domain.

AK

424,264 subscribers

70,029 views • 1 year ago •via X (Twitter)

Science & Technology Gaming

Anya Rossi• Live Now

Private livecam show

7 Comments

AK1 year ago

discuss:

GameShop Engine1 year ago

GameShop Engine. An entirely new and alternative game engine concept. One month until completion. Spread the word.

Mohan Chaitanya1 year ago

Interesting

SurfIsUp 🌊🏄🛸1 year ago

@iamarkdev @Temptranquil I know this isn't exactly similar to the @HYTOPIAgg AI SDK implementations, but made me think of you guys! Cheers!

Victor1 year ago

minecraft was just the tutorial level for ai game designers all along

Shiraz Akmal1 year ago

The font is not good

ABEL ✍️ Web3 Copywriter1 year ago

Interesting one

Related Videos

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

AK

366,858 views • 1 year ago

Google presents Genie Generative Interactive Environments introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

Google presents Genie Generative Interactive Environments introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

AK

684,273 views • 2 years ago

Meet Gemini Omni, our new model that can create anything from any input, starting with video. With Gemini Omni, you can combine images, videos and text as inputs and generate high-quality videos grounded in Gemini's real-world knowledge. #GoogleIO

Meet Gemini Omni, our new model that can create anything from any input, starting with video. With Gemini Omni, you can combine images, videos and text as inputs and generate high-quality videos grounded in Gemini's real-world knowledge. #GoogleIO

Google Gemini

87,918 views • 26 days ago

Meet Gemini Omni, our new model that can create anything from any input, starting with video. With Gemini Omni, you can combine images, videos and text as inputs and generate high-quality videos grounded in Gemini's real-world knowledge.

Meet Gemini Omni, our new model that can create anything from any input, starting with video. With Gemini Omni, you can combine images, videos and text as inputs and generate high-quality videos grounded in Gemini's real-world knowledge.

Google Gemini

32,775,488 views • 16 days ago

Generative video models are rapidly improving in quality. Meet Replay, a new AI model that can generate stunning videos from text. Replay v0.1 is designed to create ultrasmooth HD videos with a new interface. Available today for everyone. What's New? 1. Replay understands plain English prompts without prompt engineering. Try "rugged surfer" or "mermaid". 2. What's a movie without actors? Replay can crisply render close-ups of people and animals. 3. Free and fast generation from our homepage with no waiting list (link in bio). Behind the scenes, Replay is powered by aligning our new diffusion model for videos with the LLM behind Genmo Chat. We’re so excited to see what you create! Share your creations below👇

Generative video models are rapidly improving in quality. Meet Replay, a new AI model that can generate stunning videos from text. Replay v0.1 is designed to create ultrasmooth HD videos with a new interface. Available today for everyone. What's New? 1. Replay understands plain English prompts without prompt engineering. Try "rugged surfer" or "mermaid". 2. What's a movie without actors? Replay can crisply render close-ups of people and animals. 3. Free and fast generation from our homepage with no waiting list (link in bio). Behind the scenes, Replay is powered by aligning our new diffusion model for videos with the LLM behind Genmo Chat. We’re so excited to see what you create! Share your creations below👇

Genmo

154,489 views • 2 years ago

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

AK

294,442 views • 2 years ago

Google researchers just developed GameNGen, an AI that can simulate DOOM at over 20 fps It works by predicting each frame in real time with a diffusion model At scale this could mean AI will be able to create games on the fly, personalized to each player

Google researchers just developed GameNGen, an AI that can simulate DOOM at over 20 fps It works by predicting each frame in real time with a diffusion model At scale this could mean AI will be able to create games on the fly, personalized to each player

Rowan Cheung

880,195 views • 1 year ago

We are thrilled to announce the open-source release of Hallo2: a new avatar video generative model capable of generating stunning 4K resolution videos for up to 1 hour! 🎥✨ Explore the project for the open source code: Let's push the boundaries of avatar video generative model! #OpenSource #AI #FaceAnimation #GenerativeArt

Siyu ZHU

58,267 views • 1 year ago

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion Video Model, aligning them in the representation space. > Let the LLM control the VM on-the-fly. Instruction tuning maximizes the controllability. > Autoregressive LLM empowers VM to generate indefinitely long videos: Starting with a VM for 2-second videos, Pandora extends it for 8-second videos. Would #Sora+#GPT4 under Pandora produce hours-long videos? 📽️ > World Model is beyond just video generation. It’s sensory-level information processing + concept-level reasoning and reflection. Pandora bridges both, with the concept- / language-level backbone (LLM) managing & steering the sensory-level VM functionalities. 👁️🧠 Check out for a bunch of interesting results:

Super excited to introduce Pandora, a generative video World Model interactively controllable by language. #Sora and #GPT4 are both powerful. How about fusing them in a single model? 💥 Pandora gives a preview:🔭 > Build a General World Model (GWM) super efficiently by integrating pretrained autoregressive LLM and diffusion Video Model, aligning them in the representation space. > Let the LLM control the VM on-the-fly. Instruction tuning maximizes the controllability. > Autoregressive LLM empowers VM to generate indefinitely long videos: Starting with a VM for 2-second videos, Pandora extends it for 8-second videos. Would #Sora+#GPT4 under Pandora produce hours-long videos? 📽️ > World Model is beyond just video generation. It’s sensory-level information processing + concept-level reasoning and reflection. Pandora bridges both, with the concept- / language-level backbone (LLM) managing & steering the sensory-level VM functionalities. 👁️🧠 Check out for a bunch of interesting results:

Zhiting Hu

62,689 views • 2 years ago

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

Robots Digest 🤖

22,933 views • 4 months ago

AccVideo just dropped on Hugging Face Accelerating Video Diffusion Model with Synthetic Dataset present a efficient distillation method to accelerate video diffusion models with synthetic dataset method is 8.5x faster than HunyuanVideo

AccVideo just dropped on Hugging Face Accelerating Video Diffusion Model with Synthetic Dataset present a efficient distillation method to accelerate video diffusion models with synthetic dataset method is 8.5x faster than HunyuanVideo

AK

20,633 views • 1 year ago

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

📢 Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation Got only one or a few images and wondering if recovering the 3D environment is a reconstruction or generation problem? Why not do it with a generative reconstruction model! We show that a camera-conditioned video diffusion model can be transformed into a generative reconstruction model that directly outputs a high-quality 3D Gaussian Splatting representation through self-distillation, without requiring real-world training data. Check out our results in the video (wait for dynamic scenes in the second half!) : Project Page: Code and Models: Paper:

Sherwin Bahmani

66,417 views • 8 months ago

Google DeepMind just dropped 'Genie', an AI that can generate interactive video games. This is a huge deal. Genie is trained on 200,000 hours of unsupervised public internet gaming videos and can generate video games from a single prompt or image. But here's what's insane: Despite not being trained on action or text annotations, the foundation model can determine who the main character is and enable a user to control that character in the generated world. It does this through its Latent Action Model, Video Tokenizer, and Dynamics Model (will go more in-depth on this in tomorrow's newsletter for those interested). And for those asking, yes, it's research-only and not publicly available (here come the Google memes), and it does come with some limitations, like only currently creating games at 1FPS. But this is the worst AI will ever be. Anyone will be able to create their own entirely imagined virtual worlds soon, and that's a wild sentence to say out loud.

Google DeepMind just dropped 'Genie', an AI that can generate interactive video games. This is a huge deal. Genie is trained on 200,000 hours of unsupervised public internet gaming videos and can generate video games from a single prompt or image. But here's what's insane: Despite not being trained on action or text annotations, the foundation model can determine who the main character is and enable a user to control that character in the generated world. It does this through its Latent Action Model, Video Tokenizer, and Dynamics Model (will go more in-depth on this in tomorrow's newsletter for those interested). And for those asking, yes, it's research-only and not publicly available (here come the Google memes), and it does come with some limitations, like only currently creating games at 1FPS. But this is the worst AI will ever be. Anyone will be able to create their own entirely imagined virtual worlds soon, and that's a wild sentence to say out loud.

Rowan Cheung

3,376,498 views • 2 years ago

In 2026, AI world models will take the spotlight in storytelling - powering new types of interactive experiences & digital economies not seen before World models are progressing rapidly - Marble World Labs and Genie 3 Google DeepMind already generate 3D environments from text prompts, allowing users to explore them as if they were video games As creators adopt these tools, new storytelling formats will emerge. One genre I'm excited about is "generative Minecraft" - where players co-create virtual worlds together by vibe coding with world models. Game mechanics could be programmable with natural language - ex. "create a paintbrush that changes the color of anything I touch to pink" World models will also likely give rise to not just a single game, but an entire new category of generative world experiences - you could have a horror experience where you’re hiding from generated monsters, or a D&D experience where you’re roaming an infinite fantasy world with friends And with a common base model for the underlying worlds, these experiences could be inter-connected in a multiverse we could only dream about previously in science fiction A key affordance here is the role of consumers as co-creators - you can wander the multiverse as a tourist, or break out your pickaxe and become a creator anytime. This in turn would give rise to new digital economies - with creators making a living building and selling interoperable assets, serving as a guide for new players, etc The opportunity is enormous - a new category of generative worlds would not only create a new storytelling medium unlike any we’ve seen before, but also be rich training grounds for agents, robotics, and AGI If you’re excited about building new interactive experiences or the virtual economy stack with world models - we’d love to hear from you!

In 2026, AI world models will take the spotlight in storytelling - powering new types of interactive experiences & digital economies not seen before World models are progressing rapidly - Marble World Labs and Genie 3 Google DeepMind already generate 3D environments from text prompts, allowing users to explore them as if they were video games As creators adopt these tools, new storytelling formats will emerge. One genre I'm excited about is "generative Minecraft" - where players co-create virtual worlds together by vibe coding with world models. Game mechanics could be programmable with natural language - ex. "create a paintbrush that changes the color of anything I touch to pink" World models will also likely give rise to not just a single game, but an entire new category of generative world experiences - you could have a horror experience where you’re hiding from generated monsters, or a D&D experience where you’re roaming an infinite fantasy world with friends And with a common base model for the underlying worlds, these experiences could be inter-connected in a multiverse we could only dream about previously in science fiction A key affordance here is the role of consumers as co-creators - you can wander the multiverse as a tourist, or break out your pickaxe and become a creator anytime. This in turn would give rise to new digital economies - with creators making a living building and selling interoperable assets, serving as a guide for new players, etc The opportunity is enormous - a new category of generative worlds would not only create a new storytelling medium unlike any we’ve seen before, but also be rich training grounds for agents, robotics, and AGI If you’re excited about building new interactive experiences or the virtual economy stack with world models - we’d love to hear from you!

Jon Lai

29,871 views • 6 months ago

New open-source 3D world-generation model. I'm rendering a couple of worlds in the video, so check it out. You'll find the GitHub and the Hugging Face links to the model below. This is a multi-modal world model that you can use for a bunch of things: • To generate new worlds • To reconstruct worlds • To simulate 3D interactive worlds from a prompt, images, or a video You can edit the 3D outputs in Unity and Unreal Engine (they export as meshes, 3DGS files, and point clouds). You can also generate 3D characters in the world and walk around. Pretty fun stuff!

New open-source 3D world-generation model. I'm rendering a couple of worlds in the video, so check it out. You'll find the GitHub and the Hugging Face links to the model below. This is a multi-modal world model that you can use for a bunch of things: • To generate new worlds • To reconstruct worlds • To simulate 3D interactive worlds from a prompt, images, or a video You can edit the 3D outputs in Unity and Unreal Engine (they export as meshes, 3DGS files, and point clouds). You can also generate 3D characters in the world and walk around. Pretty fun stuff!

Santiago

65,308 views • 1 month ago

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6 #Robotics #AI #ZeroShot #Manipulation

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6 #Robotics #AI #ZeroShot #Manipulation

Hongyu Li

103,815 views • 8 months ago

That was quick: last week Genie 3 was revealed. One week later Skywork AI releases Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model. Matrix-Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. They open-source our model weights and codebase to advance research in interactive world modeling. Some examples:

That was quick: last week Genie 3 was revealed. One week later Skywork AI releases Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model. Matrix-Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. They open-source our model weights and codebase to advance research in interactive world modeling. Some examples:

Chubby♨️

115,138 views • 10 months ago

1/ Tencent releases HY World 1.5 (WorldPlay), an open-source real-time world model with 24fps A streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency In direct comparison to Genie 3, it's technically inferior, and there's no information on how long the consistency will last. However, Tencent has released a significant world model as open source for the first time, making it available to everyone. A few more examples, a thread 🧵

1/ Tencent releases HY World 1.5 (WorldPlay), an open-source real-time world model with 24fps A streaming video diffusion model that enables real-time, interactive world modeling with long-term geometric consistency In direct comparison to Genie 3, it's technically inferior, and there's no information on how long the consistency will last. However, Tencent has released a significant world model as open source for the first time, making it available to everyone. A few more examples, a thread 🧵

Chubby♨️

37,776 views • 5 months ago

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.

AK

38,416 views • 1 year ago