Loading video...

Video Failed to Load

Go Home

A world model controlling a drone flying outdoors at 100Hz! ๐Ÿš - Cuts position-tracking error 26-38% and attitude error up to 54% versus predictive baselines - Trained in simulation and deployed zero-shot on hardware. - predicts future dynamics in latent space instead of forecasting raw states, dodging the compounding-error...

vai_viswanathan's profile picture

Vai Viswanathan

1,877 subscribers

52,773 views โ€ข 3 months ago โ€ขvia X (Twitter)

LIVE

Anya Rossiโ€ข Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Meet VLA-JEPA, the first world model policy in LeRobot ๐ŸŒ Most VLAs just learn observation โ†’ action. VLA-JEPA also learns action-relevant dynamics: during training, a JEPA world model has to predict upcoming frames in latent space from the model's own actions. Then, at inference, the world model disappears entirely, leaving a standard, fast VLA: Qwen backbone, action head, nothing extra. In our demo it was fine-tuned on just 13 examples and ran in real time on an NVIDIA DGX Spark. On a consumer RTX 3080, it hits 10Hz using under 6GB of VRAM. World-model supervision during training, zero extra cost at inference. That's the whole trick.
0:49

Meet VLA-JEPA, the first world model policy in LeRobot ๐ŸŒ Most VLAs just learn observation โ†’ action. VLA-JEPA also learns action-relevant dynamics: during training, a JEPA world model has to predict upcoming frames in latent space from the model's own actions. Then, at inference, the world model disappears entirely, leaving a standard, fast VLA: Qwen backbone, action head, nothing extra. In our demo it was fine-tuned on just 13 examples and ran in real time on an NVIDIA DGX Spark. On a consumer RTX 3080, it hits 10Hz using under 6GB of VRAM. World-model supervision during training, zero extra cost at inference. That's the whole trick.

LeRobot

42,389 views โ€ข 2 months ago

Google presents Genie Generative Interactive Environments introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.
0:27

Google presents Genie Generative Interactive Environments introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.

AK

684,704 views โ€ข 2 years ago

SITUATION EXPLAINED: A new generation of AI labs are betting that world models are the path to general intelligence that can interact with the real world. We had General Intuition CEO Pim de Witte on MTS to explain what world models are, and why intuitively world models offer a clearer path towards general intelligence than LLMs. For humans, text is a technology for compressing information about the unfolding, three-dimensional world around us and communicating it to other people. When LLMs reason about the world, they do so by reconstructing the world through text. World models work by modeling the three-dimensional world at the architecture level and then allowing agents to interact with that world. Unlike an LLM, which understands space and time through text, world models perform spatial and temporal reasoning natively. General Intuition hopes to use world models to create embodied AIโ€”machines that can reason and act in the real world. But other companies are also pushing the frontier in this space. Odyssey ML today announced a multi-agent world model, Agora-1. Their launch video shows four participants interacting with a real-time world model simulation of GoldenEye, a first-person shooter for the N64. A playable preview of the GoldenEye simulation is available on their site.
1:31

SITUATION EXPLAINED: A new generation of AI labs are betting that world models are the path to general intelligence that can interact with the real world. We had General Intuition CEO Pim de Witte on MTS to explain what world models are, and why intuitively world models offer a clearer path towards general intelligence than LLMs. For humans, text is a technology for compressing information about the unfolding, three-dimensional world around us and communicating it to other people. When LLMs reason about the world, they do so by reconstructing the world through text. World models work by modeling the three-dimensional world at the architecture level and then allowing agents to interact with that world. Unlike an LLM, which understands space and time through text, world models perform spatial and temporal reasoning natively. General Intuition hopes to use world models to create embodied AIโ€”machines that can reason and act in the real world. But other companies are also pushing the frontier in this space. Odyssey ML today announced a multi-agent world model, Agora-1. Their launch video shows four participants interacting with a real-time world model simulation of GoldenEye, a first-person shooter for the N64. A playable preview of the GoldenEye simulation is available on their site.

MTS

11,693 views โ€ข 4 months ago

A breakthrough in real-time video generation. As a research preview developed with NVIDIA and shared at NVIDIAGTC this week, we trained a new real-time video model running on Vera Rubin. HD videos generate instantly, with time-to-first-frame under 100ms. Unlocking an entirely new creative paradigm and bolstering the foundations of our General World Model, GWM-1. Real-time generation opens a fundamentally different design space for video models and world simulation. We're investing in co-designing our models alongside advances in hardware to keep pushing this frontier.
0:54

A breakthrough in real-time video generation. As a research preview developed with NVIDIA and shared at NVIDIAGTC this week, we trained a new real-time video model running on Vera Rubin. HD videos generate instantly, with time-to-first-frame under 100ms. Unlocking an entirely new creative paradigm and bolstering the foundations of our General World Model, GWM-1. Real-time generation opens a fundamentally different design space for video models and world simulation. We're investing in co-designing our models alongside advances in hardware to keep pushing this frontier.

Runway

1,164,245 views โ€ข 6 months ago

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isnโ€™t realism. Itโ€™s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI
0:53

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isnโ€™t realism. Itโ€™s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

Biwei Huang

15,017 views โ€ข 2 months ago

Why do even our best AI models need tens of thousands of examples to learn skills that a human picks up in a handful of tries? Solving this problem is one of the great open challenges in modern AI. World models, which give AI an internal simulation of its environment, are one of the most promising paths forward. In this episode of Decoded, YC's Ankit Gupta and Francois Chaubard discuss the intuition and math behind world models, new research, and current applications in self-driving, robotics, and more. 01:45 โ€” What would perfect efficiency look like? 05:10 โ€” World models in the human brain 09:20 โ€” Control theory & the drone example 14:30 โ€” When physics breaks down 17:45 โ€” Chess, Go & the action space problem 24:10 โ€” Why AlphaGo can't scale 28:00 โ€” Monte Carlo tree search explained 34:00 โ€” Self-Driving: state space is infinite 40:30 โ€” Model-Free vs. Model-Based RL 44:00 โ€” Why robotics is the hardest case 48:20 โ€” World models that actually work 54:10 โ€” JEPA & latent space tricks 59:00 โ€” Open problems remaining 1:04:30 โ€” Does this pass the squint test?
1:14:26

Why do even our best AI models need tens of thousands of examples to learn skills that a human picks up in a handful of tries? Solving this problem is one of the great open challenges in modern AI. World models, which give AI an internal simulation of its environment, are one of the most promising paths forward. In this episode of Decoded, YC's Ankit Gupta and Francois Chaubard discuss the intuition and math behind world models, new research, and current applications in self-driving, robotics, and more. 01:45 โ€” What would perfect efficiency look like? 05:10 โ€” World models in the human brain 09:20 โ€” Control theory & the drone example 14:30 โ€” When physics breaks down 17:45 โ€” Chess, Go & the action space problem 24:10 โ€” Why AlphaGo can't scale 28:00 โ€” Monte Carlo tree search explained 34:00 โ€” Self-Driving: state space is infinite 40:30 โ€” Model-Free vs. Model-Based RL 44:00 โ€” Why robotics is the hardest case 48:20 โ€” World models that actually work 54:10 โ€” JEPA & latent space tricks 59:00 โ€” Open problems remaining 1:04:30 โ€” Does this pass the squint test?

Y Combinator

112,008 views โ€ข 2 months ago

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen ๐Ÿฅถ Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71ร— decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):
0:39

Chinese robotics company Astribot released their latest World-Action Model (WAM), Lumo-2. Technical breakdown: - based on a frozen ๐Ÿฅถ Qwen-3.5 4B VLM - trained in 3 progressive stages: 1. Action is aligned with latent world dynamics (an abstract representation of action). Real-world actions are anchored to physical constraints, while the latent space is guided to focus on motion-relevant changes. This bidirectional relationship makes the model physically grounded -> critical for a world model. 2. Action is aligned with vision and language. Reusing the vision backbone and action encoder from the frozen VLM, the authors add a custom vocabulary (for new actions), a semantic module, an action decoder, and an action projector. This aligns the (new) action representations with the (existing) vision-language semantic space. Most importantly: it builds a direct mapping from natural-language instructions to motor execution. 3. End-to-end training on language, video, and robot data. Only the new modules (everything outside the frozen backbone) are trained end-to-end across temporal reasoning, physical understanding, long-horizon, and dexterous manipulation. At the end of the day, Lumo-2 is not the best on benchmarks, but that's not the point. What's genuinely new: - a way to combine latent world modeling and action generation through progressive alignment - a physically-grounded latent dynamics space - it lifts performance on unseen objects using un-annotated human egocentric video + Vision Pro captures, no special transfer algorithm needed Why it matters: - the whole model is thin trainable adapters (semantic module, action decoder/projector) on a frozen 4B backbone (cheap) - that scale is suited for real-time embedded inference (~2.71ร— decode speedup, no accuracy loss) - its real moat is long-horizon execution, where the added temporal memory pays off far more than on any other task As a result, this robot can now make your latte (5x sped up video):

Lรฉo

32,331 views โ€ข 2 months ago

Today weโ€™re releasing V-JEPA, a method for teaching machines to understand and model the physical world by watching videos. This work is another important step towards Yann LeCunโ€™s outlined vision of AI models that use a learned understanding of the world to plan, reason and accomplish complex tasks. Details โžก๏ธ We're releasing a collection of V-JEPA vision models trained with a feature prediction objective using self-supervised learning. The models are able to understand and predict what is going on in a video, even with limited information. It learns by predicting missing or obscured parts of a video in its internal feature space. Unlike generative approaches that fill in missing pixels, this flexible approach enables up to 6x improvements in training and sample efficiency. The models were pre-trained on entirely unlabeled data, and a small amount of labeled data can be used to train a task-specific prediction head on top after pre-training. Our results show that, using a frozen backbone, our top V-JEPA models achieve 82.0% on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet1K โ€” competitive with or exceeding previous leading video models. We believe that this work is an important milestone on the path to advancing machine intelligence.
1:07

Today weโ€™re releasing V-JEPA, a method for teaching machines to understand and model the physical world by watching videos. This work is another important step towards Yann LeCunโ€™s outlined vision of AI models that use a learned understanding of the world to plan, reason and accomplish complex tasks. Details โžก๏ธ We're releasing a collection of V-JEPA vision models trained with a feature prediction objective using self-supervised learning. The models are able to understand and predict what is going on in a video, even with limited information. It learns by predicting missing or obscured parts of a video in its internal feature space. Unlike generative approaches that fill in missing pixels, this flexible approach enables up to 6x improvements in training and sample efficiency. The models were pre-trained on entirely unlabeled data, and a small amount of labeled data can be used to train a task-specific prediction head on top after pre-training. Our results show that, using a frozen backbone, our top V-JEPA models achieve 82.0% on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet1K โ€” competitive with or exceeding previous leading video models. We believe that this work is an important milestone on the path to advancing machine intelligence.

AI at Meta

704,009 views โ€ข 2 years ago

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!
51:25

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!

RoboPapers

16,760 views โ€ข 23 days ago

I am stocked to announce that I won the OpenAI Developers Codex x Mollie Hacka Worldwide Hackathon in Paris. 60+ builders, every one of us working solo, one day to ship. I built mine around a single question: who gets to own intelligence? The default answer is scary. You hand your data to a handful of labs, they train the model, they own it, and you rent back a thin slice of what your own data made possible. That is the bargain on the table today. I do not accept it. So I built Lensemble: a Tapestry like distributed training platform for JEPA based World Models. What does it enable: World Models that a community improves together, keeps sovereign, and co-owns. Two bets sit underneath it. First, the paradigm. Language models predict the next token. Powerful for text, a dead end for the physical world. A robot does not need to autocomplete sentences, it needs to predict what happens next in the world. That is what JEPA does: it learns by predicting representations instead of pixels or tokens. I am convinced world models are the most underrated paradigm in AI right now, and the closest thing we have to a ChatGPT moment for robotics. Second, the politics. Your raw trajectories never leave your machine. Each participant trains locally against a shared protocol and ships only an update, never the data. A federated round folds those updates into one shared world model, a LeWorldModel based model, and the gain is measured, not claimed: a 12k-parameter adapter on a frozen backbone, held-out prediction error down about 12 percent, the model measurably less surprised by the world. Then the upside is split by contribution weight, so the people who improved the model own a share of what it earns. This is the thesis behind Project Tapestry, the AI Alliance and Yann LeCun's push for federated, sovereign frontier AI, carried into world models and robotics. Call it Tapestry for the physical world. All of it built solo, in a single day, with Codex as my pair the whole way. Thank you to OpenAI Codex and Mollie for backing builders who ship real things, and to Boris and the organizing crew for the room and the standard you set. Intelligence the world improves, and the world owns. That is the future I want for my kids, and the one I will keep building.
1:13

I am stocked to announce that I won the OpenAI Developers Codex x Mollie Hacka Worldwide Hackathon in Paris. 60+ builders, every one of us working solo, one day to ship. I built mine around a single question: who gets to own intelligence? The default answer is scary. You hand your data to a handful of labs, they train the model, they own it, and you rent back a thin slice of what your own data made possible. That is the bargain on the table today. I do not accept it. So I built Lensemble: a Tapestry like distributed training platform for JEPA based World Models. What does it enable: World Models that a community improves together, keeps sovereign, and co-owns. Two bets sit underneath it. First, the paradigm. Language models predict the next token. Powerful for text, a dead end for the physical world. A robot does not need to autocomplete sentences, it needs to predict what happens next in the world. That is what JEPA does: it learns by predicting representations instead of pixels or tokens. I am convinced world models are the most underrated paradigm in AI right now, and the closest thing we have to a ChatGPT moment for robotics. Second, the politics. Your raw trajectories never leave your machine. Each participant trains locally against a shared protocol and ships only an update, never the data. A federated round folds those updates into one shared world model, a LeWorldModel based model, and the gain is measured, not claimed: a 12k-parameter adapter on a frozen backbone, held-out prediction error down about 12 percent, the model measurably less surprised by the world. Then the upside is split by contribution weight, so the people who improved the model own a share of what it earns. This is the thesis behind Project Tapestry, the AI Alliance and Yann LeCun's push for federated, sovereign frontier AI, carried into world models and robotics. Call it Tapestry for the physical world. All of it built solo, in a single day, with Codex as my pair the whole way. Thank you to OpenAI Codex and Mollie for backing builders who ship real things, and to Boris and the organizing crew for the room and the standard you set. Intelligence the world improves, and the world owns. That is the future I want for my kids, and the one I will keep building.

abdel

20,191 views โ€ข 3 months ago

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. Weโ€™re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robotsโ€”allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 โžก๏ธ As we continue working toward our goal of achieving advanced machine intelligence (AMI), weโ€™re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks โžก๏ธ
2:03

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. Weโ€™re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robotsโ€”allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 โžก๏ธ As we continue working toward our goal of achieving advanced machine intelligence (AMI), weโ€™re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks โžก๏ธ

AI at Meta

310,676 views โ€ข 1 year ago

Introducing ProvableWorldModel: the first provable JEPA world model. A real pretrained world model runs, and anyone can verify, on a laptop CPU, or on a mobile phone, that the committed model ran exactly as claimed, in milliseconds. The whole industry spent this week asking one question about Anthropic's Fable 5: which model actually answered me? Nobody could prove it. Not during the silent downgrades, not after the apology. The fix is a notification you still have to trust. I've been building the alternative. The intuition is simple. World models are small. Small enough to run in exact integer arithmetic, where every operation is reproducible bit for bit, attention included. The prover commits to its full execution trace before any challenge is derived. The verifier fingerprints the heavy matmuls with 1979-era randomized algebra (Freivalds) and replays everything else exactly. Change one number anywhere and the proof dies. Receipts: โ†’ real pretrained JEPA predictor (LeWorldModel), 10 MiB int8, real expert episode โ†’ inference 49 ms, verification 29 ms, CPU only, no floats, no GPU โ†’ soundness error โ‰ˆ 2^-44 โ†’ forge a single matmul output and the verifier throws FreivaldsCheckFailed Why it matters: LLM agents need ever-thicker invisible harnesses to behave, and harnesses can't be audited from the outside. World models invert that. Small, structured, exact: verifiable by construction. Intelligence you can check, not intelligence you have to trust. Don't trust, verify.
0:30

Introducing ProvableWorldModel: the first provable JEPA world model. A real pretrained world model runs, and anyone can verify, on a laptop CPU, or on a mobile phone, that the committed model ran exactly as claimed, in milliseconds. The whole industry spent this week asking one question about Anthropic's Fable 5: which model actually answered me? Nobody could prove it. Not during the silent downgrades, not after the apology. The fix is a notification you still have to trust. I've been building the alternative. The intuition is simple. World models are small. Small enough to run in exact integer arithmetic, where every operation is reproducible bit for bit, attention included. The prover commits to its full execution trace before any challenge is derived. The verifier fingerprints the heavy matmuls with 1979-era randomized algebra (Freivalds) and replays everything else exactly. Change one number anywhere and the proof dies. Receipts: โ†’ real pretrained JEPA predictor (LeWorldModel), 10 MiB int8, real expert episode โ†’ inference 49 ms, verification 29 ms, CPU only, no floats, no GPU โ†’ soundness error โ‰ˆ 2^-44 โ†’ forge a single matmul output and the verifier throws FreivaldsCheckFailed Why it matters: LLM agents need ever-thicker invisible harnesses to behave, and harnesses can't be audited from the outside. World models invert that. Small, structured, exact: verifiable by construction. Intelligence you can check, not intelligence you have to trust. Don't trust, verify.

abdel

15,377 views โ€ข 3 months ago

๐Ÿ”ฅReally excited to see the release of PAN world model, a project I had been working over the past years. PAN is a general world model capable of simulating physical, agentic, and nested worlds, synthesizing infinite interactive experiences for training AI agents. Building on top of pretrained LLMs and video diffusion models, PAN connects language, perception, action, and latent thoughts, for long-horizon simulation and reasoning. PAN shows overwhelming performance gains over JEPA-2, Cosmos-2, and other prior models. More in the thread๐Ÿ‘‡ ... 1/
0:54

๐Ÿ”ฅReally excited to see the release of PAN world model, a project I had been working over the past years. PAN is a general world model capable of simulating physical, agentic, and nested worlds, synthesizing infinite interactive experiences for training AI agents. Building on top of pretrained LLMs and video diffusion models, PAN connects language, perception, action, and latent thoughts, for long-horizon simulation and reasoning. PAN shows overwhelming performance gains over JEPA-2, Cosmos-2, and other prior models. More in the thread๐Ÿ‘‡ ... 1/

Zhiting Hu

31,252 views โ€ข 10 months ago

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 views โ€ข 7 months ago

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as โ€œLangChain for world modelsโ€. The problem is that โ€œworld modelโ€ has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48ร— faster, and runs on commodity hardware. WorldForge wires the loop: policy โ†’ candidates โ†’ score โ†’ select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.
16:27

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as โ€œLangChain for world modelsโ€. The problem is that โ€œworld modelโ€ has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48ร— faster, and runs on commodity hardware. WorldForge wires the loop: policy โ†’ candidates โ†’ score โ†’ select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.

abdel

17,001 views โ€ข 5 months ago

Andrew Ng at Stanford: โ€œStop waiting for the perfect model. Build the iteration "loop" and "graphs" around it.โ€ move from prompt โ†’ loop โ†’ graph โ†’ self-improving system โ€ข 00:05 - why every AI project needs an iteration loop โ€ข 04:08 - the full cycle: data โ†’ model โ†’ deploy โ†’ monitor โ€ข 16:02 - why faster iteration can decide who wins โ€ข 26:09 - using error analysis to improve the right data โ€ข 39:39 - why shipping a simple system beats waiting for perfection โ€ข 54:26 - monitoring drift and improving models after deployment 67-minute Stanford lecture, and itโ€™s one of the clearest playbooks on why experienced AI engineers optimize the loop around the model, not just the model itself. It's watch today, then read the full step by step roadmap in the article below
1:06:59

Andrew Ng at Stanford: โ€œStop waiting for the perfect model. Build the iteration "loop" and "graphs" around it.โ€ move from prompt โ†’ loop โ†’ graph โ†’ self-improving system โ€ข 00:05 - why every AI project needs an iteration loop โ€ข 04:08 - the full cycle: data โ†’ model โ†’ deploy โ†’ monitor โ€ข 16:02 - why faster iteration can decide who wins โ€ข 26:09 - using error analysis to improve the right data โ€ข 39:39 - why shipping a simple system beats waiting for perfection โ€ข 54:26 - monitoring drift and improving models after deployment 67-minute Stanford lecture, and itโ€™s one of the clearest playbooks on why experienced AI engineers optimize the loop around the model, not just the model itself. It's watch today, then read the full step by step roadmap in the article below

Morlex

22,861 views โ€ข 1 month ago

Odyssey is now previewing Odyssey-3, a world model that it calls a Physics Agent! The model was designed to learn how the world behaves from visual observation and then apply that understanding to control both real and simulated systems. Odyssey-3 can run across different machines and games, with a small amount of task-specific experience teaching it each set of controls. The preview covers humanoids, robot arms, a car and a drone being controlled by an AI model, playable video games, and training runs for other AI agents.
0:30

Odyssey is now previewing Odyssey-3, a world model that it calls a Physics Agent! The model was designed to learn how the world behaves from visual observation and then apply that understanding to control both real and simulated systems. Odyssey-3 can run across different machines and games, with a small amount of task-specific experience teaching it each set of controls. The preview covers humanoids, robot arms, a car and a drone being controlled by an AI model, playable video games, and training runs for other AI agents.

๐Ÿšจ AI News | TestingCatalog

23,779 views โ€ข 12 days ago

AI for robotics has some monumental challenges, like high dimensionality and reasoning with the physical world. Ashok Elluswamy, AI lead at Tesla, thinks the hardest problem is Evaluation. - Loss isn't a perfect indicator of Policy Neural Network's quality. - Testing in the real world for all the long-tail failure cases is not possible. To solve the evaluation challenge, Tesla has developed a World Simulator Model. It is trained on state and action pairs, so for a given current camera feed and current action, the model generates the next frame in the cameras. Once the world simulation is good enough, it can be hooked up to the policy network for a closed-loop simulation. In this setup, the model generates the next video frames, and based on those, the policy network comes up with the next action, which feeds back into the world simulation; this continuous loop creates a coherent simulation. The world simulation neural network is not just applicable to self-driving but also to humanoids, because Tesla trains it on common data. The examples show Optimus navigating or manipulating inside the world simulation, with all the pixels being generated by the world simulation model. The robotic agents can be tested in an accurately reproduced edge case and in other potential variations of the same edge case.
1:29

AI for robotics has some monumental challenges, like high dimensionality and reasoning with the physical world. Ashok Elluswamy, AI lead at Tesla, thinks the hardest problem is Evaluation. - Loss isn't a perfect indicator of Policy Neural Network's quality. - Testing in the real world for all the long-tail failure cases is not possible. To solve the evaluation challenge, Tesla has developed a World Simulator Model. It is trained on state and action pairs, so for a given current camera feed and current action, the model generates the next frame in the cameras. Once the world simulation is good enough, it can be hooked up to the policy network for a closed-loop simulation. In this setup, the model generates the next video frames, and based on those, the policy network comes up with the next action, which feeds back into the world simulation; this continuous loop creates a coherent simulation. The world simulation neural network is not just applicable to self-driving but also to humanoids, because Tesla trains it on common data. The examples show Optimus navigating or manipulating inside the world simulation, with all the pixels being generated by the world simulation model. The robotic agents can be tested in an accurately reproduced edge case and in other potential variations of the same edge case.

The Humanoid Hub

17,065 views โ€ข 7 months ago

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.
0:50

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

Jim Fan

567,404 views โ€ข 1 year ago

We set out to collect 10M hours of on-embodiment data as a starting point for robotics models, but couldnโ€™t find a robot reliable enough at a price that made it possible. So, we built one. Meet Pandroid: an accessible hardware platform for AI. Small enough to safely experiment with, durable enough to run in fleets. Connect it to any model or coding agent over SSH to let it interact with the world. Join the waitlist:
0:30

We set out to collect 10M hours of on-embodiment data as a starting point for robotics models, but couldnโ€™t find a robot reliable enough at a price that made it possible. So, we built one. Meet Pandroid: an accessible hardware platform for AI. Small enough to safely experiment with, durable enough to run in fleets. Connect it to any model or coding agent over SSH to let it interact with the world. Join the waitlist:

Pantograph

127,997 views โ€ข 1 month ago