Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

With enough data, robots and AI can learn “world models” that let them predict the results of their actions. These models are a way to learn how embodied AI agents can perform a wide variety of useful tasks — but they require a huge amount of data. The team...

RoboPapers's profile picture

RoboPapers

5,693 subscribers

85,927 Aufrufe • vor 7 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Training robot foundation models faces two key hurdles: how to get enough data to train an effective model, and how to make sure that new skills can be acquired quickly. The team at Rhoda AI believes that the answer is training Direct Video Action models from web data. Web data is plentiful, to the point where Rhoda can train their base model on hundreds of years of video data. And then, with the addition of robot data, they can quickly adapt it to new tasks with as little as 20 hours of in-domain data, performing complex, multi-step manipulation tasks with their purpose-built video foundation model. Tongzhou Mu 🤖🦾🦿 Eric Chan and Changan Chen joined us to talk more about their approach. Watch Episode #79 of RoboPapers, with Michael Cho - Rbt/Acc, Chris Paxton, and Jiafei Duan, to learn more!
1:09:10

Training robot foundation models faces two key hurdles: how to get enough data to train an effective model, and how to make sure that new skills can be acquired quickly. The team at Rhoda AI believes that the answer is training Direct Video Action models from web data. Web data is plentiful, to the point where Rhoda can train their base model on hundreds of years of video data. And then, with the addition of robot data, they can quickly adapt it to new tasks with as little as 20 hours of in-domain data, performing complex, multi-step manipulation tasks with their purpose-built video foundation model. Tongzhou Mu 🤖🦾🦿 Eric Chan and Changan Chen joined us to talk more about their approach. Watch Episode #79 of RoboPapers, with Michael Cho - Rbt/Acc, Chris Paxton, and Jiafei Duan, to learn more!

RoboPapers

24,352 Aufrufe • vor 1 Monat

Spatial understanding is important to moving around in complex environments and is a huge part of the challenge of generalizing to new scenes. Most world models, however, largely ignore this spatial dimension, focusing on 2D images. Not PointWorld, though. PointWorld is a 3D world model trained from real and simulated data which can perform a wide variety of manipulation tasks on a real robot, including grasping or handling articulated objects, all without any additional fine tuning. Wenlong Huang joins us to tell us more about what makes this work and how it’s different from other world models. Watch Episode #83 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
1:22:44

Spatial understanding is important to moving around in complex environments and is a huge part of the challenge of generalizing to new scenes. Most world models, however, largely ignore this spatial dimension, focusing on 2D images. Not PointWorld, though. PointWorld is a 3D world model trained from real and simulated data which can perform a wide variety of manipulation tasks on a real robot, including grasping or handling articulated objects, all without any additional fine tuning. Wenlong Huang joins us to tell us more about what makes this work and how it’s different from other world models. Watch Episode #83 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

16,197 Aufrufe • vor 1 Monat

Pretraining is essential for good performance on a wide variety of robotics tasks, and so most vision-language-action models build off of a vision language model (VLM) trained on a wide variety of image-language data. But how does the choice of VLM translate to downstream robotics performance? Jianke Zhang and @GYanjiang join us to talk about this key part of the robot policy, looking at a wide variety of different VLMs and how they perform. Interestingly, they see that performance on auxiliary tasks like quesiton answering did not lead to downstream improvements in control. To learn more, watch episode 65 of RoboPapers now, with Chris Paxton and Jiafei Duan!
1:03:46

Pretraining is essential for good performance on a wide variety of robotics tasks, and so most vision-language-action models build off of a vision language model (VLM) trained on a wide variety of image-language data. But how does the choice of VLM translate to downstream robotics performance? Jianke Zhang and @GYanjiang join us to talk about this key part of the robot policy, looking at a wide variety of different VLMs and how they perform. Interestingly, they see that performance on auxiliary tasks like quesiton answering did not lead to downstream improvements in control. To learn more, watch episode 65 of RoboPapers now, with Chris Paxton and Jiafei Duan!

RoboPapers

23,905 Aufrufe • vor 3 Monaten

Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence Tutor is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz, and Joe McCalmon, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
1:01:03

Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence Tutor is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz, and Joe McCalmon, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

36,041 Aufrufe • vor 28 Tagen

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)
1:00

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 Aufrufe • vor 1 Jahr

We’ve all seen videos of humanoid robots performing single tasks that are very impressive, like dancing or karate. But training humanoid robots to perform a wide range of complex motions is difficult. GMT is a general-purpose policy which can learn a wide range of robot motions. Watch Episode #32 of RoboPapers, with Zixuan Chen , co-hosted by Michael Cho - Rbt/Acc and Chris Paxton , to learn more.
1:03:17

We’ve all seen videos of humanoid robots performing single tasks that are very impressive, like dancing or karate. But training humanoid robots to perform a wide range of complex motions is difficult. GMT is a general-purpose policy which can learn a wide range of robot motions. Watch Episode #32 of RoboPapers, with Zixuan Chen , co-hosted by Michael Cho - Rbt/Acc and Chris Paxton , to learn more.

RoboPapers

16,605 Aufrufe • vor 9 Monaten

Yann LeCun (Chief AI Scientist, Meta, Yann LeCun), Pim de Witte (CEO, General Intuition), and Aude Durand (Kyutai, Aude Durand), talk about world models, embodied agents, Yann's new company, and the limitations of LLMs 0:00 - Introduction to World Models 5:00 - Why World Models, Intuition & Introducing Yann's new company 10:00 - Architectures + Merging Language & Interaction Data towards General Agents 20:00 - Open Source, Sovereign AI & kyutai Partnership Keynote for #aiPULSE2025 at Station F in Paris 🇫🇷
29:20

Yann LeCun (Chief AI Scientist, Meta, Yann LeCun), Pim de Witte (CEO, General Intuition), and Aude Durand (Kyutai, Aude Durand), talk about world models, embodied agents, Yann's new company, and the limitations of LLMs 0:00 - Introduction to World Models 5:00 - Why World Models, Intuition & Introducing Yann's new company 10:00 - Architectures + Merging Language & Interaction Data towards General Agents 20:00 - Open Source, Sovereign AI & kyutai Partnership Keynote for #aiPULSE2025 at Station F in Paris 🇫🇷

General Intuition

111,114 Aufrufe • vor 7 Monaten

Robots, unfortunately, tend to be expensive. And finding a robot that’s both capable of performing a wide variety of mobile manipulation tasks, and is affordable and “hackable”, is extremely difficult. Many different problems need to be addressed, from arm control to navigation to integrating your data collection strategy into hardware design. This can make it difficult for all but the most well-funded teams to “scale” real-world robotics research. Fortunately, the team behind Build Your Own Robot has a solution. Manan Anjaria, Mahi Shafiullah 🏠🤖,Jeff Cui, and Enes Erciyes joined us to talk about how they build a fully open-source mobile manipulator out of off-the-shelf parts, which has humanlike range of motion, and can perform a wide variety of tasks, all while being only roughly $10,000 to build. Watch Episode 71 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, today to learn more!
1:01:11

Robots, unfortunately, tend to be expensive. And finding a robot that’s both capable of performing a wide variety of mobile manipulation tasks, and is affordable and “hackable”, is extremely difficult. Many different problems need to be addressed, from arm control to navigation to integrating your data collection strategy into hardware design. This can make it difficult for all but the most well-funded teams to “scale” real-world robotics research. Fortunately, the team behind Build Your Own Robot has a solution. Manan Anjaria, Mahi Shafiullah 🏠🤖,Jeff Cui, and Enes Erciyes joined us to talk about how they build a fully open-source mobile manipulator out of off-the-shelf parts, which has humanlike range of motion, and can perform a wide variety of tasks, all while being only roughly $10,000 to build. Watch Episode 71 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, today to learn more!

RoboPapers

34,151 Aufrufe • vor 2 Monaten

Amjad Masad says true AGI means an AI that can enter new situations and quickly learn how to achieve goals Even the latest models still struggle with tasks outside their training data. The real breakthrough will come when AI can train itself -- like AlphaGo playing billions of games against itself
1:50

Amjad Masad says true AGI means an AI that can enter new situations and quickly learn how to achieve goals Even the latest models still struggle with tasks outside their training data. The real breakthrough will come when AI can train itself -- like AlphaGo playing billions of games against itself

Haider.

163,410 Aufrufe • vor 1 Jahr

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!
54:52

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

27,567 Aufrufe • vor 4 Monaten

Introducing General Intuition and our $133.7M Seed from Khosla Ventures, General Catalyst, and Raine. We build foundation models and general agents for environments that require deep spatial and temporal reasoning.
0:27

Introducing General Intuition and our $133.7M Seed from Khosla Ventures, General Catalyst, and Raine. We build foundation models and general agents for environments that require deep spatial and temporal reasoning.

General Intuition

2,280,589 Aufrufe • vor 8 Monaten

Is the future of AI $NEAR … ? OpenAI is racing toward a $1T IPO, quickly becoming a centralized beast of epic proportions. But what happens to your data in that world?? As chronear explains, “We can't let centralized actors determine how they store or how they use user data. It should be the users who actually control the custody or use of their data." NEAR AI is building a truly open AI, open source, where users control their data and store their data locally. What AI do you use daily? And would you switch if a better decentralized option existed?
2:34

Is the future of AI $NEAR … ? OpenAI is racing toward a $1T IPO, quickly becoming a centralized beast of epic proportions. But what happens to your data in that world?? As chronear explains, “We can't let centralized actors determine how they store or how they use user data. It should be the users who actually control the custody or use of their data." NEAR AI is building a truly open AI, open source, where users control their data and store their data locally. What AI do you use daily? And would you switch if a better decentralized option existed?

Bri Teresi

25,101 Aufrufe • vor 3 Monaten

How can we build a general-purpose “foundation model” for robot motion? Zhengyi “Zen” Luo joins us to talk about SONIC, which uses motion tracking as a foundational task for humanoid robot control, and scales humanoid control training to 9k GPU hours and 100 million frames worth of data. The result: a model with a generally-useful embedding space that can be controlled by a VLA, or from human video, to perform a wide variety of humanoid whole-body-control tasks, including with zero-shot transfer to previously unseen motions. Watch episode 72 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!
59:44

How can we build a general-purpose “foundation model” for robot motion? Zhengyi “Zen” Luo joins us to talk about SONIC, which uses motion tracking as a foundational task for humanoid robot control, and scales humanoid control training to 9k GPU hours and 100 million frames worth of data. The result: a model with a generally-useful embedding space that can be controlled by a VLA, or from human video, to perform a wide variety of humanoid whole-body-control tasks, including with zero-shot transfer to previously unseen motions. Watch episode 72 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

RoboPapers

24,802 Aufrufe • vor 2 Monaten

Zoox is making autonomous robotaxis a reality with the power of the #AWS cloud. With AWS, Zoox can build cloud-based data lakes to store & process the petabytes of sensor data required to effectively train their #MachineLearning models. Learn more. ➡️
0:59

Zoox is making autonomous robotaxis a reality with the power of the #AWS cloud. With AWS, Zoox can build cloud-based data lakes to store & process the petabytes of sensor data required to effectively train their #MachineLearning models. Learn more. ➡️

Amazon Web Services

21,510 Aufrufe • vor 1 Jahr

AI can be a powerful force for good in the world, helping us solve a wide variety of humanitarian and environmental issues. Learn more ↓
2:00

AI can be a powerful force for good in the world, helping us solve a wide variety of humanitarian and environmental issues. Learn more ↓

Google

274,947 Aufrufe • vor 2 Jahren

We're making one of the world's most data-rich organizations more accessible with Football AI Pro, a customized AI assistant that can read vast amounts of @FIFA data to deliver information to players, coaches, and fans in seconds. Learn more:
1:46

We're making one of the world's most data-rich organizations more accessible with Football AI Pro, a customized AI assistant that can read vast amounts of @FIFA data to deliver information to players, coaches, and fans in seconds. Learn more:

Lenovo

4,034,664 Aufrufe • vor 5 Monaten

Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video. Elvis Nava joins us to talk about Mimic Robotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM. Watch Episode #81 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!
1:08:43

Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video. Elvis Nava joins us to talk about Mimic Robotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM. Watch Episode #81 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

RoboPapers

46,190 Aufrufe • vor 1 Monat

Bob McGrew (Head of Research OpenAI) explains why proprietary data no longer provides companies with a competitive advantage in the AI era. Finance companies once believed their years of accumulated data would give them an edge. They planned to train specialized models on top of GPT or Llama using their exclusive information. The results shocked them. Their industry-specific models performed worse than the next generation of general purpose models. The ability to synthesize new information proved more valuable than memorizing old data. McGrew introduces the concept of "embodied labor" - the human work behind data collection. Companies spent years having employees call customers, analyze case studies, and gather information through manual processes. This accumulated knowledge required massive time and money to build. It represented thousands of hours of human effort that companies thought couldn't be replicated by competitors. But AI changes everything. Instead of years of customer calls, AI can conduct comprehensive surveys instantly. Rather than manual case analysis, AI processes thousands of examples in hours. The core insight is that value wasn't in the data itself but in the labor required to collect it. Since AI makes that labor essentially free, the advantage disappears. Companies can no longer rely on their proprietary data as a protective moat. Any competitor can use AI to replicate years of data gathering almost instantly.
1:24

Bob McGrew (Head of Research OpenAI) explains why proprietary data no longer provides companies with a competitive advantage in the AI era. Finance companies once believed their years of accumulated data would give them an edge. They planned to train specialized models on top of GPT or Llama using their exclusive information. The results shocked them. Their industry-specific models performed worse than the next generation of general purpose models. The ability to synthesize new information proved more valuable than memorizing old data. McGrew introduces the concept of "embodied labor" - the human work behind data collection. Companies spent years having employees call customers, analyze case studies, and gather information through manual processes. This accumulated knowledge required massive time and money to build. It represented thousands of hours of human effort that companies thought couldn't be replicated by competitors. But AI changes everything. Instead of years of customer calls, AI can conduct comprehensive surveys instantly. Rather than manual case analysis, AI processes thousands of examples in hours. The core insight is that value wasn't in the data itself but in the labor required to collect it. Since AI makes that labor essentially free, the advantage disappears. Companies can no longer rely on their proprietary data as a protective moat. Any competitor can use AI to replicate years of data gathering almost instantly.

Aish

240,611 Aufrufe • vor 1 Jahr

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.
44:56

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

RoboPapers

88,996 Aufrufe • vor 8 Monaten

World models — action-conditioned predictive models of the environment — are an exciting are of research for robots that can be useful both for training and for test-time compute. But video-based world models waste a lot of predictive power on reconstructing pixels, which makes model and data requirements much higher and limits how far out into the future their predictions remain viable. Instead, what if we learned a purely semantic world model, one which predicts which properties will be true about the world after a sequence of actions, without reconstructing the whole images? Jacob Berg tells us more. Watch Episode #53 of RoboPapers now, with Michael Cho - Rbt/Acc and Chris Paxton!
1:05:25

World models — action-conditioned predictive models of the environment — are an exciting are of research for robots that can be useful both for training and for test-time compute. But video-based world models waste a lot of predictive power on reconstructing pixels, which makes model and data requirements much higher and limits how far out into the future their predictions remain viable. Instead, what if we learned a purely semantic world model, one which predicts which properties will be true about the world after a sequence of actions, without reconstructing the whole images? Jacob Berg tells us more. Watch Episode #53 of RoboPapers now, with Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

39,322 Aufrufe • vor 6 Monaten