Loading video...

Video Failed to Load

Go Home

D2E Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI

_akhaliq's profile picture

AK

510,718 subscribers

25,117 views • 11 months ago •via X (Twitter)

Science & Technology
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵
1:42

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Dyna Robotics

1,789,757 views • 1 month ago

LingBot-Video is out on Hugging Face MoE-based video foundation model built for embodied intelligence 30B params, only 3B active at inference Augmented with 70K hours of embodied data on top of large-scale internet video pretraining
1:02

LingBot-Video is out on Hugging Face MoE-based video foundation model built for embodied intelligence 30B params, only 3B active at inference Augmented with 70K hours of embodied data on top of large-scale internet video pretraining

AK

40,846 views • 2 months ago

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel
3:44

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

Galbot

40,796 views • 4 months ago

Can instruction tuning be used in robotics?🚨Excited to release LLARVA, our new Vision-Action Instruction-tuned LMM for robotics! Key aspects: • Perform vision-action instruction tuning • Align vision and action modalities • Pretrained on robotic instruction data Berkeley AI Research
1:00

Can instruction tuning be used in robotics?🚨Excited to release LLARVA, our new Vision-Action Instruction-tuned LMM for robotics! Key aspects: • Perform vision-action instruction tuning • Align vision and action modalities • Pretrained on robotic instruction data Berkeley AI Research

Roei Herzig

10,517 views • 2 years ago

here's Jeff Dean talking about how labs will do multi-epoch pretraining with heavy regularization to keep scaling even with limited data. no wonder slowrun gets so much attention from pretraining teams at big labs. pretraining is about to look very very different.
0:21

here's Jeff Dean talking about how labs will do multi-epoch pretraining with heavy regularization to keep scaling even with limited data. no wonder slowrun gets so much attention from pretraining teams at big labs. pretraining is about to look very very different.

Samip

127,273 views • 5 months ago

Excited to introduce 𝐋𝐀𝐏𝐀: the first unsupervised pretraining method for Vision-Language-Action models. Outperforms SOTA models trained with ground-truth actions 30x more efficient than conventional VLA pretraining 📝: 🧵 1/9
0:58

Excited to introduce 𝐋𝐀𝐏𝐀: the first unsupervised pretraining method for Vision-Language-Action models. Outperforms SOTA models trained with ground-truth actions 30x more efficient than conventional VLA pretraining 📝: 🧵 1/9

Joel Jang

46,108 views • 1 year ago

Introducing Axis Dataset V1 - the simulation dataset for scalable robot manipulation. Can noisy, crowdsourced simulation data support embodied pretraining? Yes—with enough coverage and diversity. Continual pretraining on AXIS dataset V1 lifts π0.5 from 83.9% to 88.8% on LIBERO-Plus, with performance improving consistently as the pretraining data scales from 25% to 100% of the full dataset, showing no clear saturation. Built with researchers from Georgia Tech UC Berkeley Texas A&M University Johns Hopkins University Penn University of Michigan NUS NTU Singapore. ➡️Related links: Paper: Dataset: Project Page: GitHub Codebase:
0:34

Introducing Axis Dataset V1 - the simulation dataset for scalable robot manipulation. Can noisy, crowdsourced simulation data support embodied pretraining? Yes—with enough coverage and diversity. Continual pretraining on AXIS dataset V1 lifts π0.5 from 83.9% to 88.8% on LIBERO-Plus, with performance improving consistently as the pretraining data scales from 25% to 100% of the full dataset, showing no clear saturation. Built with researchers from Georgia Tech UC Berkeley Texas A&M University Johns Hopkins University Penn University of Michigan NUS NTU Singapore. ➡️Related links: Paper: Dataset: Project Page: GitHub Codebase:

Axis Robotics

96,538 views • 1 month ago

Today we open-source LingBot-Video — the first MoE-based video foundation model built for embodied intelligence. 🔹30B params, only 3B active at inference. 🔹Augmented with 70K hours of embodied data on top of large-scale internet video pretraining. 🔹Already outperforming Wan2.6, Seedance 1.5 Pro, and Cosmos3 Super on RBench. 🧵👇
1:32

Today we open-source LingBot-Video — the first MoE-based video foundation model built for embodied intelligence. 🔹30B params, only 3B active at inference. 🔹Augmented with 70K hours of embodied data on top of large-scale internet video pretraining. 🔹Already outperforming Wan2.6, Seedance 1.5 Pro, and Cosmos3 Super on RBench. 🧵👇

Robbyant

1,182,795 views • 2 months ago

Embodied intelligence can’t afford bad data. Perle Labs is building AI data infrastructure designed for this reality: → Human-verified, expert-validated work → On-chain auditability → Sovereign and enterprise-grade design Built to benefit everyone.
0:35

Embodied intelligence can’t afford bad data. Perle Labs is building AI data infrastructure designed for this reality: → Human-verified, expert-validated work → On-chain auditability → Sovereign and enterprise-grade design Built to benefit everyone.

Perle Labs

2,693,184 views • 7 months ago

What happens when vision🤝 robotics meet? Happy to share our new work on Pretraining Robotic Foundational Models!🔥 ARM4R is an Autoregressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better robotic model. Berkeley AI Research😊
0:49

What happens when vision🤝 robotics meet? Happy to share our new work on Pretraining Robotic Foundational Models!🔥 ARM4R is an Autoregressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better robotic model. Berkeley AI Research😊

Roei Herzig

62,805 views • 1 year ago

Simulation is a key enabler for embodied AI. It allows us to massively scale up data collection in order to experience years in minutes. Nvidia's Isaac Lab allows for rapid learning of new skills in just a few days, which directly transfer to the real world.
0:46

Simulation is a key enabler for embodied AI. It allows us to massively scale up data collection in order to experience years in minutes. Nvidia's Isaac Lab allows for rapid learning of new skills in just a few days, which directly transfer to the real world.

RIVR

25,295 views • 1 year ago

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!
1:07:18

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

RoboPapers

25,853 views • 27 days ago

1/ Can AI scaling continue through 2030? We examine whether constraints on power, chip manufacturing, training data, or data center latencies might hinder AI growth. Our analysis suggests that AI scaling can likely continue its current trend through 2030.
1:00

1/ Can AI scaling continue through 2030? We examine whether constraints on power, chip manufacturing, training data, or data center latencies might hinder AI growth. Our analysis suggests that AI scaling can likely continue its current trend through 2030.

Epoch AI

1,507,995 views • 2 years ago

April 2026 R&D footage from Luwu. Built with Raspberry Pi 5 + FEETECH serial bus servos, exploring expressive movement, reinforcement learning, and desktop embodied AI. AI shouldn’t only live on a screen.
0:32

April 2026 R&D footage from Luwu. Built with Raspberry Pi 5 + FEETECH serial bus servos, exploring expressive movement, reinforcement learning, and desktop embodied AI. AI shouldn’t only live on a screen.

XGO Robot

15,133 views • 17 days ago

Introducing GEN-0, our latest 10B+ foundation model for robots ⏱️ built on Harmonic Reasoning, new architecture that can think & act seamlessly 📈 strong scaling laws: more pretraining & model size = better 🌍 unprecedented corpus of 270,000+ hrs of dexterous data Read more 👇
0:48

Introducing GEN-0, our latest 10B+ foundation model for robots ⏱️ built on Harmonic Reasoning, new architecture that can think & act seamlessly 📈 strong scaling laws: more pretraining & model size = better 🌍 unprecedented corpus of 270,000+ hrs of dexterous data Read more 👇

Generalist

499,797 views • 10 months ago

What if you could induce models to be more concise via pretraining data curation?
1:20

What if you could induce models to be more concise via pretraining data curation?

Matthew Leavitt

7,138,104 views • 2 months ago

Introducing WALL-WM, our open-source World Model for embodied AI and the next piece of our open-source robotics stack. Carving World Action Modeling at the Event Joints Read the blog: Why it matters WALL-WM shifts robot world modeling from fixed-length action chunks to event-grounded video-action pretraining. It learns around events like reaching, contact, grasping, lifting, moving, and placing, so language, vision, and action align more naturally. Why you should care WALL-WM brings together: •Event-grounded VLA pretraining •Prior-aligned video-action architecture •Wan-based video tower + randomly initialized action DiT •Multi-view perception with sight-cone masking, tube patch masking, and Camera RoPE •Event Mode for variable-length execution •Unified Mode with Staircase Decoding •DMuon for large-scale training The goal: help robots learn what physically matters, not just what happens in the next fixed slice of time. Code (coming soon): #opensource #EmbodiedAI
1:45

Introducing WALL-WM, our open-source World Model for embodied AI and the next piece of our open-source robotics stack. Carving World Action Modeling at the Event Joints Read the blog: Why it matters WALL-WM shifts robot world modeling from fixed-length action chunks to event-grounded video-action pretraining. It learns around events like reaching, contact, grasping, lifting, moving, and placing, so language, vision, and action align more naturally. Why you should care WALL-WM brings together: •Event-grounded VLA pretraining •Prior-aligned video-action architecture •Wan-based video tower + randomly initialized action DiT •Multi-view perception with sight-cone masking, tube patch masking, and Camera RoPE •Event Mode for variable-length execution •Unified Mode with Staircase Decoding •DMuon for large-scale training The goal: help robots learn what physically matters, not just what happens in the next fixed slice of time. Code (coming soon): #opensource #EmbodiedAI

X Square Robot

42,165 views • 3 months ago

Introducing the QUANXTA Zero Series, X Square Robot’s next-generation UMI data collection solution for embodied AI. The series includes three products: QUANXTA Zero-G0: VR headset + backpack + dual grippers QUANXTA Zero-G1: headband rig + dual grippers QUANXTA Zero-E0: headband rig Built for scalable embodied AI data production: Multimodal capture. Whole-body mobile manipulation. Real-robot replay. 1 ms synchronization. 100% frame-level alignment. From human demonstrations to trainable robot data. Learn more:
2:29

Introducing the QUANXTA Zero Series, X Square Robot’s next-generation UMI data collection solution for embodied AI. The series includes three products: QUANXTA Zero-G0: VR headset + backpack + dual grippers QUANXTA Zero-G1: headband rig + dual grippers QUANXTA Zero-E0: headband rig Built for scalable embodied AI data production: Multimodal capture. Whole-body mobile manipulation. Real-robot replay. 1 ms synchronization. 100% frame-level alignment. From human demonstrations to trainable robot data. Learn more:

X Square Robot

354,518 views • 2 months ago

🚀 MiniCPM enters the physical world — enabling robots to understand, remember, and act. We open-source MiniCPM-Robot, our first embodied AI model series, including: 🤖 MiniCPM-RobotManip — a 1.5B general-purpose Vision-Language-Action (VLA) model for robotic manipulation. 🐕 MiniCPM-RobotTrack — a compact model for real-world target tracking. ⚡ PhyAI — a high-performance inference framework built for embodied models. Together, they bring efficient, practical, and open embodied intelligence closer to real-world robots. ⭐ GitHub: 🤗 MiniCPM-RobotManip: 🤗 MiniCPM-RobotTrack:
1:17

🚀 MiniCPM enters the physical world — enabling robots to understand, remember, and act. We open-source MiniCPM-Robot, our first embodied AI model series, including: 🤖 MiniCPM-RobotManip — a 1.5B general-purpose Vision-Language-Action (VLA) model for robotic manipulation. 🐕 MiniCPM-RobotTrack — a compact model for real-world target tracking. ⚡ PhyAI — a high-performance inference framework built for embodied models. Together, they bring efficient, practical, and open embodied intelligence closer to real-world robots. ⭐ GitHub: 🤗 MiniCPM-RobotManip: 🤗 MiniCPM-RobotTrack:

OpenBMB

330,472 views • 1 month ago

💡Why Vana? “Ultimately, AI models are only as good as their training data. So if you want to build the best AI, you need the best data.” shares the vision behind Vana’s mission, centered on DataDAOs that aggregate specific datasets, reward data owners for their contributions and level up AI models with high-quality data🏆
0:56

💡Why Vana? “Ultimately, AI models are only as good as their training data. So if you want to build the best AI, you need the best data.” shares the vision behind Vana’s mission, centered on DataDAOs that aggregate specific datasets, reward data owners for their contributions and level up AI models with high-quality data🏆

vana

108,377 views • 1 year ago