Загрузка видео...

Не удалось загрузить видео

На главную

Cosmos Policy turns a pretrained video diffusion model into a robot controller. Instead of redesigning the architecture, it injects robot state, actions, and values directly as latent frames inside the video model

robotsdigest's profile picture

Robots Digest 🤖

5,086 subscribers

22,933 просмотров • 9 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

We release Cosmos Policy 💫: a state-of-the-art robot policy built on a video diffusion model backbone. - policy + world model + value function — in 1 model - no architectural changes to the base video model - SOTA in LIBERO (98.5%), RoboCasa (67.1%), & ALOHA tasks (93.6%) 🧵👇
1:31

We release Cosmos Policy 💫: a state-of-the-art robot policy built on a video diffusion model backbone. - policy + world model + value function — in 1 model - no architectural changes to the base video model - SOTA in LIBERO (98.5%), RoboCasa (67.1%), & ALOHA tasks (93.6%) 🧵👇

Moo Jin Kim

152,783 просмотров • 9 месяцев назад

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 просмотров • 8 месяцев назад

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:
0:51

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

Chuang Gan

21,317 просмотров • 4 месяцев назад

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.
2:05

A humanoid robot policy trained solely on synthetic data generated by a world model. Research Scientist Joel Jang presents NVIDIA's DreamGen pipeline: ⦿ Post-train the world model Cosmos-Predict2 with a small set of real teleoperation demos. ⦿ Prompt the world model to generate synthetic video data with verbs and scenarios not used in the world model’s post-training. ⦿ Auto-label synthetic video data with action sequences. ⦿ Train robot policies using only synthetic data. That's it. Deploy zero-shot to a real humanoid robot.

The Humanoid Hub

20,968 просмотров • 1 год назад

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source
1:33

World Model meets robot policy! Robbyant's LingBot-VA: unifies video world modeling and robotic policy learning. - A single model generates both future video and the actions to make it real. - Long-term memory enables long-horizon tasks. - Claims significant outperformance over π₀.₅ in real-world tasks. - It's open-source

The Humanoid Hub

17,721 просмотров • 9 месяцев назад

Today, we are releasing a research preview of Rho-1, our 19B omni model that understands and generates text, images, video and robot actions, all inside a single neural network.
1:00

Today, we are releasing a research preview of Rho-1, our 19B omni model that understands and generates text, images, video and robot actions, all inside a single neural network.

Reka

161,522 просмотров • 27 дней назад

S1 learns new tasks like a language model. You prompt it with a video demonstration, and it outputs robot actions to complete the task in any environment and in any embodiment.
0:28

S1 learns new tasks like a language model. You prompt it with a video demonstration, and it outputs robot actions to complete the task in any environment and in any embodiment.

Skild AI

308,754 просмотров • 2 месяцев назад

VLAs? WAMs? Are language and video models the right foundations for robotics? Introducing Grounded Action Model (GAM): a new paradigm that builds robot foundation models on top of a pretrained 3D grounding model. Ground first. Then learn to act. 🧵👇
0:40

VLAs? WAMs? Are language and video models the right foundations for robotics? Introducing Grounded Action Model (GAM): a new paradigm that builds robot foundation models on top of a pretrained 3D grounding model. Ground first. Then learn to act. 🧵👇

Jiafei Duan

693,344 просмотров • 1 месяц назад

Diffusion has shown great promise for generating robot **actions**, can it act as a **world model** to generate the future conditioned on actions? In our work led by han qi Haocheng Yin and in collaboration with Yilun Du, we show a **controllable** action-conditioned video diffusion model can produce photorealistic and (near) physics-accurate future predictions. This ability strengthens the policy via: - ranking different action proposals and selecting the best, or - **visual** trajectory optimization by optimizing the action proposals using gradient ascent. Learn more about Generative Predictive Control (GPC) at:
1:37

Diffusion has shown great promise for generating robot **actions**, can it act as a **world model** to generate the future conditioned on actions? In our work led by han qi Haocheng Yin and in collaboration with Yilun Du, we show a **controllable** action-conditioned video diffusion model can produce photorealistic and (near) physics-accurate future predictions. This ability strengthens the policy via: - ranking different action proposals and selecting the best, or - **visual** trajectory optimization by optimizing the action proposals using gradient ascent. Learn more about Generative Predictive Control (GPC) at:

Heng Yang

38,572 просмотров • 1 год назад

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048 abs: project page:
1:38

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048 abs: project page:

AK

718,894 просмотров • 3 лет назад

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄
0:32

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

23,580 просмотров • 3 месяцев назад

We introduce W.A.L.T, a diffusion model for photorealistic video generation. Our model is a transformer trained on image and video generation in a shared latent space. 🧵👇
0:43

We introduce W.A.L.T, a diffusion model for photorealistic video generation. Our model is a transformer trained on image and video generation in a shared latent space. 🧵👇

Agrim Gupta

431,392 просмотров • 2 лет назад

Today mimic and friends are excited to share mimic-video, a new class of Video-Action Model that elevates video model backbones as first class citizens for robot learning!
2:07

Today mimic and friends are excited to share mimic-video, a new class of Video-Action Model that elevates video model backbones as first class citizens for robot learning!

Elvis Nava

88,707 просмотров • 10 месяцев назад

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks
1:51

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

Yinghao Xu

47,703 просмотров • 9 месяцев назад

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.
2:32

We’re excited to share DiT4DiT, an end-to-end Video-Action Model for robot learning that unifies a video Diffusion Transformer and an action Diffusion Transformer in a single cascaded framework. By leveraging the rich spatiotemporal and physical dynamics learned through video generation, rather than static image-text priors, DiT4DiT achieves state-of-the-art results on LIBERO (98.6%) and RoboCasa GR1 (50.8%) with far less training data, delivering over 10× better sample efficiency and up to 7× faster convergence. Real-world deployment on a humanoid robot further shows robust generalization. We believe this is a step toward making video generation a powerful backbone for robot policy learning. This work builds upon the brilliant foundations laid by Nvidia's GR00T and Cosmos. Project: Paper: Code: Coming soon. In the meantime, you can ask your coding agent to reproduce the method based on GR00T/Cosmos.

Shuo Yang

31,872 просмотров • 7 месяцев назад

We're excited to introduce AW-0: Agentic World Action Models! 🤖 (0/8) The first World Action Model that plans in code, predicts the future, and controls the robot—all in one generative model. AW-0 jointly generates: 🧠 Code - explicit skill calls, object arguments, and control flow 🎥 Future video - predictive visual dynamics 🦾 Robot actions - continuous motor control All three are generated together through joint denoising, letting semantic plans directly shape both predicted futures and robot behavior, with: 🤝 Hybrid execution — generated programs can also run directly through perception and control APIs ⚡ Just 40 ms of additional inference latency over the base WAM on an RTX 5090 📚 Pretrained on ~3,000 hours of robot + human demonstrations One model for planning, prediction, and execution. 🌐 Project: 📄 Paper: 💻 Code: 🤗 Model: 🧵 How it works—and what we found ↓
0:46

We're excited to introduce AW-0: Agentic World Action Models! 🤖 (0/8) The first World Action Model that plans in code, predicts the future, and controls the robot—all in one generative model. AW-0 jointly generates: 🧠 Code - explicit skill calls, object arguments, and control flow 🎥 Future video - predictive visual dynamics 🦾 Robot actions - continuous motor control All three are generated together through joint denoising, letting semantic plans directly shape both predicted futures and robot behavior, with: 🤝 Hybrid execution — generated programs can also run directly through perception and control APIs ⚡ Just 40 ms of additional inference latency over the base WAM on an RTX 5090 📚 Pretrained on ~3,000 hours of robot + human demonstrations One model for planning, prediction, and execution. 🌐 Project: 📄 Paper: 💻 Code: 🤗 Model: 🧵 How it works—and what we found ↓

trevordarrell

25,163 просмотров • 5 дней назад

🌍 Introducing Mundo, led by my student Yihang Luo: a customized world model that turns images of a place into an environment for exploration, editing, and simulation. Explore new viewpoints, change objects and physical properties, or simulate robot navigation and manipulation — all within the captured environment. The key is an explicit, editable world state: code specifies actions, physics computes their consequences, and a video model renders the outcome using the input images as visual context. Project: Blog: Code coming soon!
2:22

🌍 Introducing Mundo, led by my student Yihang Luo: a customized world model that turns images of a place into an environment for exploration, editing, and simulation. Explore new viewpoints, change objects and physical properties, or simulate robot navigation and manipulation — all within the captured environment. The key is an explicit, editable world state: code specifies actions, physics computes their consequences, and a video model renders the outcome using the input images as visual context. Project: Blog: Code coming soon!

Chen Change (Cavan) Loy

10,616 просмотров • 9 дней назад

This looks like another step toward removing the data bottleneck in robotics. The bottleneck exists because recording demonstrations is easy, but labeling every grasp, hold, move, and release is slow and expensive. Perceptron Egocentric dropped a new robotics video annotation system. You give it raw robot-camera or first-person/egocentric video. It gives back structured labels that look more like training data: where each small manipulation starts and ends, what the subtask is, what each hand is doing, where the hands are frame by frame, and which hand is left or right. So, Input: raw video of a person or robot doing a task. Output: a machine-readable breakdown of the physical actions inside that video. Basically, it is a machine that converts robot video into policy-training supervision. That means timestamps, subtask boundaries, per-hand actions, and left-right hand grounding. So this will turn large piles of robot or egocentric video into supervision for training robot policies.
0:58

This looks like another step toward removing the data bottleneck in robotics. The bottleneck exists because recording demonstrations is easy, but labeling every grasp, hold, move, and release is slow and expensive. Perceptron Egocentric dropped a new robotics video annotation system. You give it raw robot-camera or first-person/egocentric video. It gives back structured labels that look more like training data: where each small manipulation starts and ends, what the subtask is, what each hand is doing, where the hands are frame by frame, and which hand is left or right. So, Input: raw video of a person or robot doing a task. Output: a machine-readable breakdown of the physical actions inside that video. Basically, it is a machine that converts robot video into policy-training supervision. That means timestamps, subtask boundaries, per-hand actions, and left-right hand grounding. So this will turn large piles of robot or egocentric video into supervision for training robot policies.

Rohan Paul

18,903 просмотров • 3 месяцев назад

Instead of choosing between training a world model and training a language conditioned robot policy, why not do both? LDA-1B s a new foundation model that is trained on 30,000 hours of human and robot interaction data. Part of the secret is that LDA-1B jointly learns forward dynamics, action prediction, and visual forecasting, all in a structured DINO latent space which avoids the pitfalls of redundant pixel-level prediction which isn’t necessarily aligned robot action. This approach works on both dexterous hands and simple robot grippers; it also generalizes across objects, tasks, and scenes. Jiangran Lyu joins us to explain. Learn more on Episode 98 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton!
42:33

Instead of choosing between training a world model and training a language conditioned robot policy, why not do both? LDA-1B s a new foundation model that is trained on 30,000 hours of human and robot interaction data. Part of the secret is that LDA-1B jointly learns forward dynamics, action prediction, and visual forecasting, all in a structured DINO latent space which avoids the pitfalls of redundant pixel-level prediction which isn’t necessarily aligned robot action. This approach works on both dexterous hands and simple robot grippers; it also generalizes across objects, tasks, and scenes. Jiangran Lyu joins us to explain. Learn more on Episode 98 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

16,345 просмотров • 2 месяцев назад