Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Agentic Real2Sim is an interesting take on world modeling. Instead of learning physics directly from videos, it lets a Vision-Language Agent reconstruct a physics simulator from a single real-world video. The result is an editable, executable world model that can be queried, simulated, and improved.

Robots Digest 🤖

5,994 subscribers

12,075 views • 2 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

5 Comments

Robots Digest 🤖2 months ago

The pipeline is surprisingly modular. • VLMs estimate scene geometry, objects, materials, and physical properties • These are assembled into a physics simulator • The simulator is validated against the original video • If it diverges, the VLM iteratively debugs and refines the scene It's essentially an "agent in the loop" for simulator construction.

Robots Digest 🤖2 months ago

What stands out is the shift in philosophy. Instead of training another neural world model, Agentic Real2Sim treats simulation as a reasoning problem. The VLM acts like an engineer—observing reality, writing a simulator, testing it against evidence, fixing mistakes, and repeating until the simulated world matches the real one.

Robots Digest 🤖2 months ago

paper: website:

Matthew Nicoletti2 months ago

Reconstructing a queryable physics sim from one video is a clever approach.

Dr. Richard2 months ago

This application of vision language agents to physics based world modeling is compelling. Reconstructing an editable simulator from a single video delivers the real world efficiency I look for in advanced manufacturing R&D. Excellent work! 🎯

Related Videos

Can we learn a 3D world model that predicts object dynamics directly from videos? Introducing Particle-Grid Neural Dynamics: a learning-based simulator for deformable objects that trains from real-world videos. Website: ArXiv: Code: Demo: To appear at #RSS2025

Can we learn a 3D world model that predicts object dynamics directly from videos? Introducing Particle-Grid Neural Dynamics: a learning-based simulator for deformable objects that trains from real-world videos. Website: ArXiv: Code: Demo: To appear at #RSS2025

Kaifeng Zhang

46,516 views • 1 year ago

Odyssey is now previewing Odyssey-3, a world model that it calls a Physics Agent! The model was designed to learn how the world behaves from visual observation and then apply that understanding to control both real and simulated systems. Odyssey-3 can run across different machines and games, with a small amount of task-specific experience teaching it each set of controls. The preview covers humanoids, robot arms, a car and a drone being controlled by an AI model, playable video games, and training runs for other AI agents.

Odyssey is now previewing Odyssey-3, a world model that it calls a Physics Agent! The model was designed to learn how the world behaves from visual observation and then apply that understanding to control both real and simulated systems. Odyssey-3 can run across different machines and games, with a small amount of task-specific experience teaching it each set of controls. The preview covers humanoids, robot arms, a car and a drone being controlled by an AI model, playable video games, and training runs for other AI agents.

🚨 AI News | TestingCatalog

23,779 views • 29 days ago

How can agents understand the world from diverse language? 🌎 Excited to introduce Dynalang, an agent that learns to understand language by 𝙢𝙖𝙠𝙞𝙣𝙜 𝙥𝙧𝙚𝙙𝙞𝙘𝙩𝙞𝙤𝙣𝙨 𝙖𝙗𝙤𝙪𝙩 𝙩𝙝𝙚 𝙛𝙪𝙩𝙪𝙧𝙚 with a multimodal world model!

How can agents understand the world from diverse language? 🌎 Excited to introduce Dynalang, an agent that learns to understand language by 𝙢𝙖𝙠𝙞𝙣𝙜 𝙥𝙧𝙚𝙙𝙞𝙘𝙩𝙞𝙤𝙣𝙨 𝙖𝙗𝙤𝙪𝙩 𝙩𝙝𝙚 𝙛𝙪𝙩𝙪𝙧𝙚 with a multimodal world model!

Jessy Lin

107,579 views • 3 years ago

Been seeing a lot of Astra for real2sim, but what you need for robotics: physics and affordance. Our SOTA recipe specifically solves for: • Affordance: model object details that convey functionality. • Physics: calibrate simulation dynamics against real-world observations. Our agent has the action space of a “full-stack simulation engineer” and jointly considers geometry, materials, colliders, articulation, and physics. It automatically calibrates physics against real trajectories and matches task-specific dynamics. Simulation environments is useful only if it can provide meaningful policy level signals. Methods, comparisons, and results below.

Been seeing a lot of Astra for real2sim, but what you need for robotics: physics and affordance. Our SOTA recipe specifically solves for: • Affordance: model object details that convey functionality. • Physics: calibrate simulation dynamics against real-world observations. Our agent has the action space of a “full-stack simulation engineer” and jointly considers geometry, materials, colliders, articulation, and physics. It automatically calibrates physics against real trajectories and matches task-specific dynamics. Simulation environments is useful only if it can provide meaningful policy level signals. Methods, comparisons, and results below.

Sharon Lee

119,796 views • 9 days ago

Introducing Project Eden, a world model research preview from VAST AI Research Project Eden is a persistent, multiplayer world model that fundamentally breaks from existing paradigms by decoupling the underlying world state from visual rendering. Instead of treating the world as a sequence of transient frames, Eden treats it as a structured, evolving environment that runs continuously, can be modified by user actions, and can be consistently observed from any viewpoint.

Introducing Project Eden, a world model research preview from VAST AI Research Project Eden is a persistent, multiplayer world model that fundamentally breaks from existing paradigms by decoupling the underlying world state from visual rendering. Instead of treating the world as a sequence of transient frames, Eden treats it as a structured, evolving environment that runs continuously, can be modified by user actions, and can be consistently observed from any viewpoint.

Tripo

3,395,968 views • 4 months ago

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

AK

85,570 views • 2 years ago

A video world model can make a convincing clip and still get the physics wrong. Our researchers just released Physis-Lang, an open self-evolving framework that adds physics reasoning to video captions. The captions explain why and how a scene unfolds. We use them to fine-tune world models and add that reasoning to prompts when generating video. Adding physics reasoning to the prompt alone improved NVIDIA Cosmos 3’s PhyGenBench score by 5.62 points without retraining. Paper and project:

A video world model can make a convincing clip and still get the physics wrong. Our researchers just released Physis-Lang, an open self-evolving framework that adds physics reasoning to video captions. The captions explain why and how a scene unfolds. We use them to fine-tune world models and add that reasoning to prompts when generating video. Adding physics reasoning to the prompt alone improved NVIDIA Cosmos 3’s PhyGenBench score by 5.62 points without retraining. Paper and project:

NVIDIA AI

31,266 views • 14 days ago

Introducing Starchild-1 from Odyssey, the first ever real-time multimodal world model. This a model that can generate interactive simulations of the world that you can—for the first time ever—hear. Starchild-1 represents a big step towards a general-purpose world simulator.

Introducing Starchild-1 from Odyssey, the first ever real-time multimodal world model. This a model that can generate interactive simulations of the world that you can—for the first time ever—hear. Starchild-1 represents a big step towards a general-purpose world simulator.

Oliver Cameron

247,184 views • 4 months ago

Announcing Arcway ⊹ ࣪ ˖ A real-time 3D engine where anyone can design a home. It is a simulated 3D world where buyers explore, change, and decide as light, physics, building rules, and real products move as one.

Announcing Arcway ⊹ ࣪ ˖ A real-time 3D engine where anyone can design a home. It is a simulated 3D world where buyers explore, change, and decide as light, physics, building rules, and real products move as one.

Caleb Barclay

283,687 views • 10 months ago

At Avalon we are building "Real-time creating" - the ability to generate gameplay ready persistent worlds prompted from text. While others are building real-time video world models, Avalon is building real-time world generation inside a fully playable, persistent multiplayer engine. Internally running at 3840×2180 at 60 FPS. Built on Unreal Engine. Multiplayer by default. Persistent by default. Gameplay-ready by default. This is not a video latent replay. Not a simulation of interaction. It is a real 3D world with physics, logic, and authoritative multiplayer state. Avalon is trained on proprietary Avalon interaction data and powered by a hybrid system that combines language understanding, 3D model generation, procedural systems, and structured gameplay logic synthesis. Players can walk through a live world and generate environments, assets, mechanics, and entirely new gameplay modes using natural language. We accomplish this through a combination of 3D model generation, game logic generation based on our proprietary systems, and AI driven world creation. While other players are inside it. Changes persist instantly. State is synchronized in real time. Creation happens inside the world, not outside of it. Describe a biome. Spawn a civilization. Create a survival mode. Build a dungeon crawler. Launch a new game inside the world. Avalon interprets intent and integrates it directly into the live multiplayer environment. This is not a world model predicting video. This is a gameplay engine that understands language. If you can describe it, you can build it. And others can walk into it instantly.

At Avalon we are building "Real-time creating" - the ability to generate gameplay ready persistent worlds prompted from text. While others are building real-time video world models, Avalon is building real-time world generation inside a fully playable, persistent multiplayer engine. Internally running at 3840×2180 at 60 FPS. Built on Unreal Engine. Multiplayer by default. Persistent by default. Gameplay-ready by default. This is not a video latent replay. Not a simulation of interaction. It is a real 3D world with physics, logic, and authoritative multiplayer state. Avalon is trained on proprietary Avalon interaction data and powered by a hybrid system that combines language understanding, 3D model generation, procedural systems, and structured gameplay logic synthesis. Players can walk through a live world and generate environments, assets, mechanics, and entirely new gameplay modes using natural language. We accomplish this through a combination of 3D model generation, game logic generation based on our proprietary systems, and AI driven world creation. While other players are inside it. Changes persist instantly. State is synchronized in real time. Creation happens inside the world, not outside of it. Describe a biome. Spawn a civilization. Create a survival mode. Build a dungeon crawler. Launch a new game inside the world. Avalon interprets intent and integrates it directly into the live multiplayer environment. This is not a world model predicting video. This is a gameplay engine that understands language. If you can describe it, you can build it. And others can walk into it instantly.

AVALON

66,204 views • 8 months ago

The next frontier after language models is world models, and learning to predict the next state of a world. Hear from our researchers how these models really work at a low-level and the mountain we're climbing. A general-purpose world simulator (AKA sci-fi) is now in sight.

The next frontier after language models is world models, and learning to predict the next state of a world. Hear from our researchers how these models really work at a low-level and the mountain we're climbing. A general-purpose world simulator (AKA sci-fi) is now in sight.

Odyssey

25,961 views • 10 months ago

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

The Humanoid Hub

17,721 views • 8 months ago

How far can 5,500 hours of real-world driving data take you? All the way to VATIX, a new open-source, state-of-the-art world model for driving video generation. 🚗 The first major result from the NATIX × Valeo Group collaboration is here!

How far can 5,500 hours of real-world driving data take you? All the way to VATIX, a new open-source, state-of-the-art world model for driving video generation. 🚗 The first major result from the NATIX × Valeo Group collaboration is here!

NATIX Network

45,724 views • 1 month ago

📁 Fei-Fei Li, one of the most influential AI scientists in the world, argues that common sense is not as mysterious as we think. Much of it is already encoded in language, distilled through thousands of years of human communication. When an AI places the sun at the top of a drawing, it does not need deep physics, it is learning shared patterns. In her view, common sense can be learned, and in many ways an AI is not so different from a five year old child.

📁 Fei-Fei Li, one of the most influential AI scientists in the world, argues that common sense is not as mysterious as we think. Much of it is already encoded in language, distilled through thousands of years of human communication. When an AI places the sun at the top of a drawing, it does not need deep physics, it is learning shared patterns. In her view, common sense can be learned, and in many ways an AI is not so different from a five year old child.

Jon Hernandez

20,744 views • 8 months ago

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,867 views • 4 months ago

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen 🥶 Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71× decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Léo

32,513 views • 2 months ago