正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Introducing 𝐃𝐫𝐞𝐚𝐦𝐆𝐞𝐧! We got humanoid robots to perform totally new 𝑣𝑒𝑟𝑏𝑠 in new environments through video world models. We believe video world models will solve the data problem in robotics. Bringing the paradigm of scaling human hours to GPU hours. Quick 🧵

Joel Jang

3,093 subscribers

117,574 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

11 条评论

Joel Jang 的头像

Joel Jang1 年前

Currently, robot data scaling is done through human labor. Recent work showed some potential signs of robots doing useful things in unseen homes (i.e., open-world generalization), but this required taking the physical robots and collecting data in 100+ homes.

Joel Jang 的头像

Joel Jang1 年前

How about new 𝑣𝑒𝑟𝑏𝑠? When prompted, current robots stand still or perform tasks they were trained on (e.g. pick-and-place). There is currently no work in literature that can enable robots to perform new verbs outside of the teleoperation data for visuomotor robot policies.

Joel Jang 的头像

Joel Jang1 年前

We introduce 𝐃𝐫𝐞𝐚𝐦𝐆𝐞𝐧, an embarrassingly simple 4-step pipeline that generates synthetic robot training data using video world models. (1) Fine-tune, (2) Prompt, (3) Extract actions, (4) Train. Simple as that.

Joel Jang 的头像

Joel Jang1 年前

Through DreamGen, we generate “Dreams” or 𝑁𝑒𝑢𝑟𝑎𝑙 𝑇𝑟𝑎𝑗𝑒𝑐𝑡𝑜𝑟𝑖𝑒𝑠 of 22 new verbs in 10 unseen environments, and train robots to perform these tasks "zero-shot".

Joel Jang 的头像

Joel Jang1 年前

DreamGen can also augment seen, contact-rich tasks that are hard to simulate (e.g. folding, scooping m&ms) for different robot systems (Franka & SO-100) and different robot policies (Diffusion Policy, π₀, GR00T N1), all resulting in non-trivial gains.

Joel Jang 的头像

Joel Jang1 年前

We also introduce 𝘋𝑟𝘦𝑎𝘮𝐺𝘦𝑛 𝐵𝘦𝑛𝘤ℎ, a video generative benchmark for robotics that shows a positive correlation with downstream robot policies, so that video model researchers can help enable robotics without actually having to set up their own physical robot systems.

Joel Jang 的头像

Joel Jang1 年前

This was an 8-month research effort, jointly led by @SeonghyeonYe @zy27962986 @szxiangjn, and advised by @scott_e_reed @yukez @DrJimFan , and with amazing collaborators from Nvidia GEAR Lab, Nvidia Cosmos Team, and UW. Check out more videos in the blog post & details in the paper! 🌐 Blog post: 📝 Paper: We will also be releasing the code & an API to try out DreamGen (prompt -> lerobot dataset) in the upcoming days! Do reach out if you are interested in joining our GR00T Dreams Team. We are scaling up the data, GPUs, and the team!

VistaShares 的头像

VistaShares1 年前

Discover the future of AI investing. AIS delivers exposure to the companies driving the next wave of innovation—semiconductors, data centers, and AI applications. Explore the supercycle today.

Thomas Farfeleder 的头像

Thomas Farfeleder1 年前

@threadreaderapp unroll

JulianSaks 的头像

JulianSaks1 年前

So cool to see this! Congrats 🔥

Raymond Yu 的头像

Raymond Yu1 年前

Congrats Joel! Doing big things👀

相关视频

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Today we are introducing Dyna-2, a world-action model pre-trained on one million hours of human video. At this scale, for the first time, we discovered several new scaling laws: • world-action models exhibit scaling law on human data across four orders of magnitude, from 1000 to 1,000,000 hours, • this human data scaling law implied a scaling law on never seen robot data, • both data and objective matter; world modeling and scaling on video data are essential for cross-embodiment scaling transfer to emerge 🧵

Dyna Robotics

1,776,346 次观看 • 23 天前

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

76,421 次观看 • 1 个月前

Can robot foundation models scale? Xiaomi-Robotics-1 explores this question with over 100,000 hours of real-world manipulation data. Pre-trained on large-scale real-world trajectories and post-trained with cross-embodiment robot data, Xiaomi-Robotics-1 demonstrates consistent scaling across data and model size, strong generalization in unseen environments, and efficient adaptation to new tasks. 🔗 #Robotics #EmbodiedAI #FoundationModels #RobotLearning #XiaomiAI #XiaomiRobotics

Can robot foundation models scale? Xiaomi-Robotics-1 explores this question with over 100,000 hours of real-world manipulation data. Pre-trained on large-scale real-world trajectories and post-trained with cross-embodiment robot data, Xiaomi-Robotics-1 demonstrates consistent scaling across data and model size, strong generalization in unseen environments, and efficient adaptation to new tasks. 🔗 #Robotics #EmbodiedAI #FoundationModels #RobotLearning #XiaomiAI #XiaomiRobotics

Xiaomi Tech

22,591 次观看 • 1 个月前

🧵The quiet revolution in robotics isn't humanoid, or world models.. It's the rise of inspection & maintenance robots, quietly scaling across dirty, dangerous, and distant industrial environments. This sector is projected to hit $8.3B by 2030. Let's break down why it's growing so fast. [Save this thread for later 📌]

🧵The quiet revolution in robotics isn't humanoid, or world models.. It's the rise of inspection & maintenance robots, quietly scaling across dirty, dangerous, and distant industrial environments. This sector is projected to hit $8.3B by 2030. Let's break down why it's growing so fast. [Save this thread for later 📌]

Lukas Ziegler

172,280 次观看 • 3 个月前

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵

Mahi Shafiullah 🏠🤖

95,070 次观看 • 2 个月前

the micro1 robotics lab: real world data for intelligent models that co-exist in the physical world. we’re in-the-wild across 75 countries in 6,000+ unique environments collecting data. diverse movements, objects, and settings. the future of AI is as human as you can imagine. join us to start training robots today (link in comments).

the micro1 robotics lab: real world data for intelligent models that co-exist in the physical world. we’re in-the-wild across 75 countries in 6,000+ unique environments collecting data. diverse movements, objects, and settings. the future of AI is as human as you can imagine. join us to start training robots today (link in comments).

Ali Ansari

208,500 次观看 • 5 个月前

Schaeffler is bringing UK's humanoid robots from concept to factory reality. Within the next 5 years, hundreds of humanoid robots will be deployed in Schaeffler's global factories, with beta versions first implemented in 2026–2027. Schaeffler is also becoming the preferred supplier of joint actuators for the humanoid wheeled platform and is jointly exploring next-generation actuator architectures with its partner. More importantly, the two companies will collaborate deeply on data and skills: continuously accumulating real-world manufacturing data through teleoperation, synthetic data, and peripheral data acquisition robots to train AI models, enabling humanoid robots to learn faster and perform more reliably in the factory.

Schaeffler is bringing UK's humanoid robots from concept to factory reality. Within the next 5 years, hundreds of humanoid robots will be deployed in Schaeffler's global factories, with beta versions first implemented in 2026–2027. Schaeffler is also becoming the preferred supplier of joint actuators for the humanoid wheeled platform and is jointly exploring next-generation actuator architectures with its partner. More importantly, the two companies will collaborate deeply on data and skills: continuously accumulating real-world manufacturing data through teleoperation, synthetic data, and peripheral data acquisition robots to train AI models, enabling humanoid robots to learn faster and perform more reliably in the factory.

CyberRobo

19,907 次观看 • 7 个月前

Introducing General World Models. We believe the next major advancement in AI will come from systems that understand the visual world and its dynamics, which is why we’re starting a new long-term research effort around general world models. Learn more:

Introducing General World Models. We believe the next major advancement in AI will come from systems that understand the visual world and its dynamics, which is why we’re starting a new long-term research effort around general world models. Learn more:

Runway

714,468 次观看 • 2 年前

Proud to share what I’ve been working on with my colleagues at Rhoda AI: Direct Video-Action Models (DVA). TL;DR: - We pre-train causal video models from scratch to control robots - They handle complex production tasks for hours without intervention - Only use ~10 hours of robot data How? 🧵👇

Proud to share what I’ve been working on with my colleagues at Rhoda AI: Direct Video-Action Models (DVA). TL;DR: - We pre-train causal video models from scratch to control robots - They handle complex production tasks for hours without intervention - Only use ~10 hours of robot data How? 🧵👇

Tongzhou Mu 🤖🦾🦿

19,257 次观看 • 5 个月前

JUST IN: Dyna Robotics just published one of the most important research papers in robotics this year. It could fundamentally change how robot foundation models are trained. A scaling law that transfers from human video to robot performance. Dyna-2 is out and it's 🔥 Here's what that means in plain terms. Dyna-2 was pre-trained on ONE MILLION hours of egocentric human video, 170 years of continuous human experience, cooking, folding, assembling, cleaning. And as that human data scaled, robot performance improved. Predictably. Monotonically. Across 39 tasks on two different robot embodiments the model had never seen. → 1,000 hours pre-training → 20% normalised task performance → 10,000 hours → 28% → 100,000 hours → 45% → 1,000,000 hours → 53% Human video exists at effectively unlimited scale. Every cook, every factory worker, every craftsperson wearing a camera is generating training data for future robots. But the finding that stunned even the researchers, world modeling is what makes the transfer work. A model trained to predict future video AND actions massively outperforms one trained on actions alone. Video is the new scaling axis for robotics. One more jaw-dropping data point. 13 minutes of teleoperation data was enough to fine-tune Dyna-2 to open a bottle cap using two five-fingered robot hands. The robots are coming, and they're learning from us directly :D Read more here: Congrats Jason Ma and team! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

JUST IN: Dyna Robotics just published one of the most important research papers in robotics this year. It could fundamentally change how robot foundation models are trained. A scaling law that transfers from human video to robot performance. Dyna-2 is out and it's 🔥 Here's what that means in plain terms. Dyna-2 was pre-trained on ONE MILLION hours of egocentric human video, 170 years of continuous human experience, cooking, folding, assembling, cleaning. And as that human data scaled, robot performance improved. Predictably. Monotonically. Across 39 tasks on two different robot embodiments the model had never seen. → 1,000 hours pre-training → 20% normalised task performance → 10,000 hours → 28% → 100,000 hours → 45% → 1,000,000 hours → 53% Human video exists at effectively unlimited scale. Every cook, every factory worker, every craftsperson wearing a camera is generating training data for future robots. But the finding that stunned even the researchers, world modeling is what makes the transfer work. A model trained to predict future video AND actions massively outperforms one trained on actions alone. Video is the new scaling axis for robotics. One more jaw-dropping data point. 13 minutes of teleoperation data was enough to fine-tune Dyna-2 to open a bottle cap using two five-fingered robot hands. The robots are coming, and they're learning from us directly :D Read more here: Congrats Jason Ma and team! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

23,576 次观看 • 23 天前

Boston Dynamics and Google DeepMind announced a partnership to integrate Gemini Robotics AI models with the new electric Atlas humanoid robots. Got to say hi to Carolina Parada before the presentation

Boston Dynamics and Google DeepMind announced a partnership to integrate Gemini Robotics AI models with the new electric Atlas humanoid robots. Got to say hi to Carolina Parada before the presentation

The Humanoid Hub

14,689 次观看 • 8 个月前

What could scaling unlock for biology? Introducing ProGen3- our next AI foundation models for protein generation. We develop compute-optimal scaling laws up to 46B parameters on 1.5T tokens with real evidence in the wet lab. +we solve a new set of challenges for drug discovery

What could scaling unlock for biology? Introducing ProGen3- our next AI foundation models for protein generation. We develop compute-optimal scaling laws up to 46B parameters on 1.5T tokens with real evidence in the wet lab. +we solve a new set of challenges for drug discovery

Ali Madani

80,665 次观看 • 1 年前

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

27,567 次观看 • 7 个月前

What if robots, humans, and video models could all describe action in the same language? Introducing Hydra-0🐍: a generalist world model that represents robot actions as action flow. One interface. Many embodiments, tasks, and environments. 🧵1/7

What if robots, humans, and video models could all describe action in the same language? Introducing Hydra-0🐍: a generalist world model that represents robot actions as action flow. One interface. Many embodiments, tasks, and environments. 🧵1/7

Hongyu Li

52,442 次观看 • 13 天前

Models have run out of internet to learn from. The next frontier is the physical world. Every week we turn 21,000+ hours of real world activity into labeled, sensor-enriched data, continuous observation of how the physical world actually moves.

Models have run out of internet to learn from. The next frontier is the physical world. Every week we turn 21,000+ hours of real world activity into labeled, sensor-enriched data, continuous observation of how the physical world actually moves.

375ai

10,774 次观看 • 2 个月前

Asimov (Asimov) pays households and businesses worldwide to record their daily routines, generating thousands of hours a day of diverse human movement data to train humanoid robots. They've built a global network, along with the full stack, from proprietary collection hardware to annotation tooling, and are already providing data to the largest robotics companies in the world. Congrats on the launch, Lyem Ningthou and Anshul!

Asimov (Asimov) pays households and businesses worldwide to record their daily routines, generating thousands of hours a day of diverse human movement data to train humanoid robots. They've built a global network, along with the full stack, from proprietary collection hardware to annotation tooling, and are already providing data to the largest robotics companies in the world. Congrats on the launch, Lyem Ningthou and Anshul!

Y Combinator

74,650 次观看 • 5 个月前

We’re introducing intrinsic discovery, a new learning method that allows models to explore and learn from open-ended environments without human supervision. Using intrinsic discovery, we trained a world model that surpasses GPT-5.6 Sol at simulating terminal environments.

We’re introducing intrinsic discovery, a new learning method that allows models to explore and learn from open-ended environments without human supervision. Using intrinsic discovery, we trained a world model that surpasses GPT-5.6 Sol at simulating terminal environments.

Induction Labs

67,199 次观看 • 6 天前

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

Our vision is for AI that uses world models to adapt in new and dynamic environments and efficiently learn new skills. We’re sharing V-JEPA 2, a new world model with state-of-the-art performance in visual understanding and prediction. V-JEPA 2 is a 1.2 billion-parameter model, trained on video, that can enable zero-shot planning in robots—allowing them to plan and execute tasks in unfamiliar environments. Learn more about V-JEPA 2 ➡️ As we continue working toward our goal of achieving advanced machine intelligence (AMI), we’re also releasing three new benchmarks for evaluating how well existing models can reason about the physical world from video. Learn more and download the new benchmarks ➡️

AI at Meta

310,435 次观看 • 1 年前

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 次观看 • 11 个月前