Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Native Action-Prior Learning from Videos for World Action Models paper:

_akhaliq's profile picture

AK

532,959 subscribers

19,186 Aufrufe • vor 20 Tagen •via X (Twitter)

Wissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from
1:43

3D-VLA A 3D Vision-Language-Action Generative World Model Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from

AK

85,570 Aufrufe • vor 2 Jahren

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:
0:51

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

Chuang Gan

21,317 Aufrufe • vor 4 Monaten

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI
0:53

World models have a causality problem. Realistic videos are not enough. A world model should predict the future caused by an action, not just a plausible future. We find that many latent-action world models generate convincing videos while barely responding to the supplied action. The root cause lies in Latent Action Models (LAMs): reconstruction objectives entangle action-induced dynamics with action-irrelevant visual changes, producing visually confounded latent actions. Our solution is CD-LAM (Causally Debiased Latent Action Model). It cleanses latent action representations before pre-training begins, requiring zero changes to backbone architectures, latent dimensions, or action interfaces. Key Benchmark Results: - Action Controllability: Cuts action-following error by >30% while simultaneously enhancing visual fidelity. - Extreme Sample Efficiency: Achieves baseline performance in just 3,000 post-training steps instead of 50,000 - a >10x speedup. By removing visual confounding, CD-LAM bridges the gap between passive video prediction and actionable causal intelligence for robotics. The next bottleneck for world models isn’t realism. It’s causality. Paper, project page and additional resources in the reply below. #WorldModels #EmbodiedAI #Robotics #CausalAI

Biwei Huang

15,084 Aufrufe • vor 3 Monaten

Large language models reason through text. Vision‑language‑action models reason through the real world. By fusing perception, context, and action from live video, VLAs deliver the awareness physical AI needs for next‑gen robotics and edge systems.
1:28

Large language models reason through text. Vision‑language‑action models reason through the real world. By fusing perception, context, and action from live video, VLAs deliver the awareness physical AI needs for next‑gen robotics and edge systems.

Intel

15,931 Aufrufe • vor 8 Monaten

👀 Seeing is just the beginning. With Qwen-MM-Plugins, turn your favorite agent harness multimodal-native — read images, videos & documents, edit videos, work with 3D/CAD, and more. From multimodal models → multimodal agents. 🚀 Watch it in action:
4:07

👀 Seeing is just the beginning. With Qwen-MM-Plugins, turn your favorite agent harness multimodal-native — read images, videos & documents, edit videos, work with 3D/CAD, and more. From multimodal models → multimodal agents. 🚀 Watch it in action:

Qwen

799,041 Aufrufe • vor 2 Monaten

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:
1:00

Glad that our work “Inference-Time Enhancement of Generative Robot Policies via Predictive World Modeling”, led by Han Qi, has been accepted to IEEE Robotics and Automation Letters! 🎉 We propose Generative Predictive Control (GPC): sample action proposals from a pretrained diffusion policy (“look back”), roll them out with a diffusion-based action-conditioned video world model (“look forward”), then rank or optimize the actions using either a learned reward model or VLM preferences. Conceptually, this is trajectory optimization / MPC with hybrid sampling + gradient optimization, interpreted through modern diffusion priors and video world models. Interestingly, we first posted the paper on arXiv in Feb 2025, when action-conditioned video world models for planning were still rare—now this direction is rapidly gaining traction. Still many open questions, e.g., • how to avoid local minima in planning • what representations work best for world models • how to balance physics priors vs. data-driven learning Paper:

Heng Yang

19,293 Aufrufe • vor 7 Monaten

What if robots could improve themselves by learning from their own failures in the real-world? Introducing 𝗣𝗟𝗗 (𝗣𝗿𝗼𝗯𝗲, 𝗟𝗲𝗮𝗿𝗻, 𝗗𝗶𝘀𝘁𝗶𝗹𝗹) — a recipe that enables Vision-Language-Action (VLA) models to self-improve for high-precision manipulation tasks. PLD couples real-world residual reinforcement learning with standard supervised fine-tuning — letting robots discover, recover, and distill their own data flywheel. Quick 🧵
0:59

What if robots could improve themselves by learning from their own failures in the real-world? Introducing 𝗣𝗟𝗗 (𝗣𝗿𝗼𝗯𝗲, 𝗟𝗲𝗮𝗿𝗻, 𝗗𝗶𝘀𝘁𝗶𝗹𝗹) — a recipe that enables Vision-Language-Action (VLA) models to self-improve for high-precision manipulation tasks. PLD couples real-world residual reinforcement learning with standard supervised fine-tuning — letting robots discover, recover, and distill their own data flywheel. Quick 🧵

Wenli Xiao

186,346 Aufrufe • vor 1 Jahr

See machine learning in action! This demo from #TheBringUp uses the AMD Kria KV260 Vision AI Starter Kit to help visualize how AI models process inputs and generate outputs. 🎥
1:17

See machine learning in action! This demo from #TheBringUp uses the AMD Kria KV260 Vision AI Starter Kit to help visualize how AI models process inputs and generate outputs. 🎥

AMD Embedded

11,615 Aufrufe • vor 3 Monaten

PM Narendra Modi Ji continues his tough action against paper leak! 47 officers dismissed from NTA!
1:10

PM Narendra Modi Ji continues his tough action against paper leak! 47 officers dismissed from NTA!

Pradeep Bhandari(प्रदीप भंडारी)🇮🇳

12,770 Aufrufe • vor 3 Monaten

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

New paper release: Learning Agile Perceptive Traversal of Sparse 3D Structures for Humanoids Paper Videos We study how to do learning and sim2real for 3d traversal, such as monkey bars and overhanging obstacles.
2:48

New paper release: Learning Agile Perceptive Traversal of Sparse 3D Structures for Humanoids Paper Videos We study how to do learning and sim2real for 3d traversal, such as monkey bars and overhanging obstacles.

C. Zhang

83,397 Aufrufe • vor 1 Monat

Saturday Night #AEWCollision 8/7c, TNT + HBO Max Tomorrow, 2/7! @Toxic_Thekla in Action! Ahead of learning what the stipulation will be for her AEW World Championship rematch against @CallMeKrisStat, "The Toxic Spider" will be in action in Las Vegas, TOMORROW!
0:20

Saturday Night #AEWCollision 8/7c, TNT + HBO Max Tomorrow, 2/7! @Toxic_Thekla in Action! Ahead of learning what the stipulation will be for her AEW World Championship rematch against @CallMeKrisStat, "The Toxic Spider" will be in action in Las Vegas, TOMORROW!

All Elite Wrestling

11,892 Aufrufe • vor 8 Monaten

From ideas to working models! 🔧✨ Participants of Prerana Batch 24, Cycle 2 designed and laser-cut their own clocks and telescopes, turning curiosity into creation through hands-on learning and innovation. A glimpse of experiential learning in action! 🌟 #Prerana #ExperientialLearning #SchoolEducation Prerana: An Experiential Learning Program
0:53

From ideas to working models! 🔧✨ Participants of Prerana Batch 24, Cycle 2 designed and laser-cut their own clocks and telescopes, turning curiosity into creation through hands-on learning and innovation. A glimpse of experiential learning in action! 🌟 #Prerana #ExperientialLearning #SchoolEducation Prerana: An Experiential Learning Program

Ministry of Education

15,833 Aufrufe • vor 28 Tagen

Grounding World Simulation Models in a Real-World Metropolis paper:
2:04

Grounding World Simulation Models in a Real-World Metropolis paper:

AK

33,892 Aufrufe • vor 7 Monaten

VLAExplain — Interpreting Vision-Language-Action (VLA) Models VLAExplain is an interpretability toolkit designed to help users visually understand the inner workings of Vision-Language-Action (VLA) models. Currently, attention analysis is supported for both the pi05 and unifolm-vla models. For details, please check pi05 and UnifoLM-VLA readme files respectively. Demo of pi05 in action:
0:28

VLAExplain — Interpreting Vision-Language-Action (VLA) Models VLAExplain is an interpretability toolkit designed to help users visually understand the inner workings of Vision-Language-Action (VLA) models. Currently, attention analysis is supported for both the pi05 and unifolm-vla models. For details, please check pi05 and UnifoLM-VLA readme files respectively. Demo of pi05 in action:

Ryohei Sasaki@engineer

12,774 Aufrufe • vor 6 Monaten

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.
0:29

YouTube is a LARGE dataset of demonstration videos to train Generalist robot agents, but lacks action data. How can we learn DEXTEROUS skills from them? In #CoRL2024, we explore the problem of learning a Generalist Piano Playing agent from YouTube videos.

Julen Urain

42,451 Aufrufe • vor 2 Jahren

Action-controllable world models are evolving fast. LingBot-World-Infinity can run an unbroken session of real-time world generation for up to an hour. A distilled model can stream 720p at 60 fps. World models are going to be a critical piece in training robotic agents for unseen worlds.
1:33

Action-controllable world models are evolving fast. LingBot-World-Infinity can run an unbroken session of real-time world generation for up to an hour. A distilled model can stream 720p at 60 fps. World models are going to be a critical piece in training robotic agents for unseen worlds.

The Humanoid Hub

14,196 Aufrufe • vor 3 Monaten

🤖✨ What if models that take action in the physical world could think through your instructions? Meet MolmoAct, our new fully open Action Reasoning Model (ARM) that does just that. 🧵
2:18

🤖✨ What if models that take action in the physical world could think through your instructions? Meet MolmoAct, our new fully open Action Reasoning Model (ARM) that does just that. 🧵

Ai2

180,624 Aufrufe • vor 1 Jahr

What is the hypocritical world waiting for to take action?
0:27

What is the hypocritical world waiting for to take action?

Astrid

68,171 Aufrufe • vor 2 Jahren