🤖 One predictive backbone, three distinct tasks, consistent gains:... a strong signal that investing in reusable world models is the right abstraction for robot learning. Toyota Research Institute (TRI) just ran the same idea three different ways—and it worked each time. Using our NVIDIA Cosmos Predict 2–style world models, they hit SOTA with dynamic view synthesis, teleop data augmentation (even up against MimicGen as an upper bound), and navigation world models. 🙌 📚 dynamic view synthesis 📚 teleop data augmentation (also compare to MimicGen as an upper bound) 📚 navigation world models 📸 fromshow more

Protein language models just got an upgrade. Meet Profluent-E1:... a free, flexible, frontier protein sequence encoder. E1 is built with retrieval augmentation to learn from multiple sequences. Models trained over 4T tokens with only 150M-600M params, E1 is SOTA for zero-shot functional and unsupervised structural tasks. It raises the bar for protein representation learning and is freely available today.show more

What comes after vision-language-action models in robotics? 🤖 Join... NVIDIA researchers live to explore World Action Models (WAMs), Vision-Language-Action models (VLAs), and hybrid robot foundation models, and see how NVIDIA Cosmos 3 natively represents action alongside video. 🕑 Thursday, Aug. 13 • 9:00–10:00 a.m. PT 📅Add calendar:show more

Kimi K3 built a fully procedural browser-based 3D exploration... game using Three.js WebGPU and GPU compute. It procedurally generated the environment, while using a 3D asset generation tool to create the rider and horse models, producing an expansive open world with forests, a log-cabin village, snowy mountains, and dynamic weather. External assets used: animated cowboy and horse models and terrain data.show more

World models aren't the same as regular video models.... The benchmarks are different. I'm really stoked to see what visionary creators do with this technology.show more

AI Is Moving Beyond “Generating Videos” — Toward “Generating... Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:show more

Bring new robot testing environments to life with World... Labs and Isaac Sim. 🤖 If you can describe a world 🌎, you can start testing in it the same day. Learn how to: 1. Export scenes from World Labs' Marble as Gaussian splats 2. Convert to USD using NVIDIA Omniverse NuRec 3. Import into NVIDIA Isaac Sim 4. Add a robot and run the simulation Read the guide ➡️ #SIGGRAPHAsia2025show more

(1/6) X-Humanoid 🤖: Scaling up data for Humanoid Robots.... We convert human daily activity videos (from Ego-Exo4D) into humanoid videos (i.e., Tesla Optimus) performing tasks like cooking or fixing a bike. This data can be potentially used to train robot policies and world models. 🔥 Project page: Paper link:show more

JUST IN: Meta AI introduces Voicebox, an all-in-one generative... speech model. Voicebox is an impressive breakthrough! It could do for speech what other models like GPT-3 and Stable Diffusion have done for text and images. Some key details: - Voicebox can synthesize speech across 6 languages - It's a general-purpose model that can perform tasks it wasn't trained on. It can perform noise removal, content editing, style conversion, and more - Supports in-context text-to-speech synthesis and cross-lingual style transfer - It's 20x faster than current models and outperforms single-purpose models through in-context learning paper: blog:show more

made an app that guesses where you are in... the world with just a picture using image embeddings trained on street view data first time using swiftui, consumer apps in general, TestFlight belowshow more

OpenAI's Deep Research is getting a run for its... money. Deep Lake was just released, and it's a different take on an AI system that can do deep research on your own data. You can use Deep Lake to build AI search with reasoning on your private and public data. (Look at the attached videos to get an idea of how it works.) If you want to research proprietary and sensitive data, Deep Research won't help you because it's limited to public data. Deep Lake, however, will allow you to use your private data. On top of that, Deep Lake supports multi-modal retrieval from the ground up. It uses vision language models for data ingestion and retrieval so that you can connect any data (PDFs, images, videos, structured data, etc.) You can even use mixed-data queries! Deep Lake can search your data from S3, Dropbox, and GCP. It learns from your queries over time, making the results as relevant to your work as possible!show more

MoVieS Motion-Aware 4D Dynamic View Synthesis in One Second

Over the Reality just hit another major milestone in... the AI era. We have now surpassed 200,000 mappings in our dataset. 🎉 These are not just images. They are high-quality 3D maps generated from the mapping activity of our global community, which has produced more than 94M images of real-world locations so far. The dataset also includes 900TB+ of spatial data that can power Visual AI models, Robotics VPS for real-world navigation and positioning, XR VPS for precise spatial anchoring and AR experiences, Spatial Computing applications, Digital Twins, and the next generation of real-world AI infrastructure. In just the past 2 weeks, our dataset grew by +17,955 mapped locations. That is nearly +10% growth in just 14 days. More maps. More data. More demand for $OVR.show more

Is Jev is a game changer for robotics? We... gave Jev a robot body and handed it complex tasks across navigation, spatial reasoning, and world geometry 120 different real + simulated tasks and environments benchmarking performance against Dimcode, Astra, Fable, Opus, and 5.6 We graded against speed, cost, tokens, # collisions, path quality Code, Data, and Paper dropping tomorrow. The results were surprising.show more

We just hit a new huge milestone. 🏆 Thanks... to our amazing community, we have now reached 170,000 3D Maps across the world. Here are the numbers that define what we’ve built so far: ↳ 170,000 mapped locations ↳ 767 TB of world data ↳ 85.2M+ images captured ↳ And a steady, unstoppable month over month growth. A single dataset designed to help machines see, understand, and navigate space, also used to train spatial AI and visual models, and to geolocalize robots and experiences with precision through our VPS, in order to interact seamlessly with the real world. And yes... you read that right! We are already in talks with several top AI companies that want access to our dataset to train their AI models and to support what comes next, including robotics and advanced visual and video models. It is all one story, one ecosystem, one vision, and it is called Over the Reality.show more

Vision AI agents are becoming a practical way to... automatically turn video data from the physical world into operational intelligence in factories, cities, warehouses and transportation systems. See how Roboflow, DeepHow, and Linker Vision use NVIDIA Cosmos and NVIDIA Metropolis to generate synthetic data, fine-tune vision models faster, and deploy vision agents at scale. 📰 Read the blog:show more

We are sharing a major update to our General... World Models efforts. GWM Worlds 2 can generate full, interactive, real-time video simulations in one continuous 720p stream at 24 fps and audio at 48000 Hz, responding to your inputs as you explore. It is the first world model that generalizes to arbitrary actions rather than a fixed set of actions or just navigation. Dynamic actions are what actually matter for building gaming and interactive experiences, and for training agents in these simulations. We also introduced WorldPrompt, a way to tell GWM-2 that a world can be split into two kinds of state: what persists and what changes over time. For example: persistent gravity, physics, and light, alongside actions that describe movement, gestures, object interactions, speech, and sound.show more

In a masterclass at Sequoia Capital AI Ascent, Jim... Fan laid out the "Great Parallel": how robotics is speedrunning the LLM playbook. 🔹 VLA → WAM: Moving from language-heavy models to "World Action Models" that dream in physics. 🔹 Teleop → EgoScale: Replacing manual data with human egocentric video. 🔹 Simulation 2.0: Using neural simulators like DreamDojo to turn compute into environments. "Our generation was born too late to explore the earth and too early to explore the stars. But we are born just in time to solve robotics." He believes that robots will pass the Physical Turing Test in the coming 2–3 years.show more

introducing Krea for Grok bot. generate images and video... with the best models in the world, right from Grok. try it now 👇show more

Most AI models just predict the next pixel to... build a narrative or an image. Gemini Omni is different. Gemini Omni has an intuitive understanding of forces like gravity, kinetic energy, and fluid dynamics for more realistic movements that follow real-world logic.show more