Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Embodied AI is exploding, and it’s been amazing to learn about it with you all. We have now recorded over a hundred discussions with the best researchers in the world on embodied artificial intelligence and robotics, covering topics like scaling robot learning, humanoid whole body control, tactile sensors, sim-to-real,... show more

RoboPapers

6,504 subscribers

15,195 Aufrufe • vor 25 Tagen •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

2 Kommentare

Profilbild von AIme Tang

AIme Tangvor 25 Tagen

What is the real bottleneck for Embodied AI?

Profilbild von The Humanoid Hub

The Humanoid Hubvor 24 Tagen

100 episodes 🎉

Ähnliche Videos

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

RoboPapers

25,954 Aufrufe • vor 1 Monat

Most robot policies today still largely lack memory: they make all their decisions based on what they can see right now. MemER aims to change that by learning which frames are important; this lets it deal with tasks like object search. Ajay Sridhar, Jenny Pan, and @satviks107Sharma tell us about how to achieve this fundamental capability for long-horizon task execution. Watch Episode #54 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

Most robot policies today still largely lack memory: they make all their decisions based on what they can see right now. MemER aims to change that by learning which frames are important; this lets it deal with tasks like object search. Ajay Sridhar, Jenny Pan, and @satviks107Sharma tell us about how to achieve this fundamental capability for long-horizon task execution. Watch Episode #54 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

RoboPapers

17,875 Aufrufe • vor 9 Monaten

Robots need to be able to apply pressure and make contact with objects as needed in order to accomplish their tasks. From compliance to working safely around humans to whole-body manipulation of heavy objects, combining force and position control can dramatically expand the capabilities of robots. This is especially true for legged robots, which have so much ability to exert forces on the world around them. But how do we train robots which can do this? Baoxiong Jia tells us more in our discussion of his team’s recent, Best Paper Award winning work on learning a unified policy for position and force control, called UniFP. To learn more, watch Episode #49 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton.

Robots need to be able to apply pressure and make contact with objects as needed in order to accomplish their tasks. From compliance to working safely around humans to whole-body manipulation of heavy objects, combining force and position control can dramatically expand the capabilities of robots. This is especially true for legged robots, which have so much ability to exert forces on the world around them. But how do we train robots which can do this? Baoxiong Jia tells us more in our discussion of his team’s recent, Best Paper Award winning work on learning a unified policy for position and force control, called UniFP. To learn more, watch Episode #49 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton.

RoboPapers

44,825 Aufrufe • vor 9 Monaten

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

27,945 Aufrufe • vor 7 Monaten

How can we build robotic hands with truly superhuman dexterity? Daxo Robotics is developing a unique tendon-driven soft robot hand, which aims to be tougher and more capable than a traditional humanoid hand. Each finger consists of many different tendons, which act in concert to move or manipulate. This is a special episode of RoboPapers where, instead of talking about a scientific paper, we talk to Tom Zhang, founder of Daxo Robotics, in order to learn both about his background and about how this one-of-a-kind robot hand design works. Watch Episode #40 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now!

How can we build robotic hands with truly superhuman dexterity? Daxo Robotics is developing a unique tendon-driven soft robot hand, which aims to be tougher and more capable than a traditional humanoid hand. Each finger consists of many different tendons, which act in concert to move or manipulate. This is a special episode of RoboPapers where, instead of talking about a scientific paper, we talk to Tom Zhang, founder of Daxo Robotics, in order to learn both about his background and about how this one-of-a-kind robot hand design works. Watch Episode #40 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

16,782 Aufrufe • vor 10 Monaten

How can we build a general-purpose “foundation model” for robot motion? Zhengyi “Zen” Luo joins us to talk about SONIC, which uses motion tracking as a foundational task for humanoid robot control, and scales humanoid control training to 9k GPU hours and 100 million frames worth of data. The result: a model with a generally-useful embedding space that can be controlled by a VLA, or from human video, to perform a wide variety of humanoid whole-body-control tasks, including with zero-shot transfer to previously unseen motions. Watch episode 72 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

How can we build a general-purpose “foundation model” for robot motion? Zhengyi “Zen” Luo joins us to talk about SONIC, which uses motion tracking as a foundational task for humanoid robot control, and scales humanoid control training to 9k GPU hours and 100 million frames worth of data. The result: a model with a generally-useful embedding space that can be controlled by a VLA, or from human video, to perform a wide variety of humanoid whole-body-control tasks, including with zero-shot transfer to previously unseen motions. Watch episode 72 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

RoboPapers

24,802 Aufrufe • vor 5 Monaten

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

RoboPapers

30,767 Aufrufe • vor 8 Monaten

Robotics has changed dramatically over the last eight years. Ted Xiao has been involved in the cutting edge of robot learning through this period, spending those eight years at Google Brain/Google Deepmind. And he’s identified three eras of robot learning. These eras are: - The Era of Existence Proofs - trying different methods like QT-Opt, on-robot RL - The Era of Foundation Models - transitioning to data collection and clean objectives (i.e. supervised learning) - The Era of Scaling - orders of magnitude more data and larger models, enabling reasoning, long-horizon actions, and cross-embodiment transfer Watch Episode 78 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan to learn more!

Robotics has changed dramatically over the last eight years. Ted Xiao has been involved in the cutting edge of robot learning through this period, spending those eight years at Google Brain/Google Deepmind. And he’s identified three eras of robot learning. These eras are: - The Era of Existence Proofs - trying different methods like QT-Opt, on-robot RL - The Era of Foundation Models - transitioning to data collection and clean objectives (i.e. supervised learning) - The Era of Scaling - orders of magnitude more data and larger models, enabling reasoning, long-horizon actions, and cross-embodiment transfer Watch Episode 78 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan to learn more!

RoboPapers

36,838 Aufrufe • vor 4 Monaten

Full episode dropping soon! Geeking out with Jason Ma Tianyu Li Anh-Quan Pham Chet Bhateja on Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Co-hosted by Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan

Full episode dropping soon! Geeking out with Jason Ma Tianyu Li Anh-Quan Pham Chet Bhateja on Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Co-hosted by Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan

RoboPapers

10,912 Aufrufe • vor 1 Monat

Evaluating robot policies is hard. Every lab has a different robot; reproducible evaluations are really challenging. This makes it hard for us to know which methods for learning robot policies are likely to perform the best in real-world scenarios. Taking a page from LLM evaluations like Chatbot Arena, RoboArena aims to address this problem through crowdsourcing evaluations with a network of different evaluators. Watch Episode #34 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton , now to learn more from authors Pranav Atreya and Karl Pertsch!

Evaluating robot policies is hard. Every lab has a different robot; reproducible evaluations are really challenging. This makes it hard for us to know which methods for learning robot policies are likely to perform the best in real-world scenarios. Taking a page from LLM evaluations like Chatbot Arena, RoboArena aims to address this problem through crowdsourcing evaluations with a network of different evaluators. Watch Episode #34 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton , now to learn more from authors Pranav Atreya and Karl Pertsch!

RoboPapers

15,056 Aufrufe • vor 11 Monaten

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!

RoboPapers

16,760 Aufrufe • vor 23 Tagen

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

20,434 Aufrufe • vor 9 Monaten

Teaching robots from human video is an important part of overcoming the “data gap” in robotics, but many of the details still need to be worked out. Homanga Bharadhwaj tells us about two recent research papers, Gen2Act and Spider, which go over different aspects of the problem: Gen2Act uses generative video models to create a reference of how a task should be performed given a language prompt; then, it uses a multi-purpose policy that can “translate” from human video to robot motion. However, Gen2Act has its limitations, in particular when it comes to dexterous, contact-rich tasks. That’s where SPIDER comes in: it uses human data together with simulation to train policies across many different humanoid hands and datasets. Also of note is that this is our first episode with our new rotating co-host, Jiafei Duan. To learn more, watch Episode #57 of RoboPapers now, with Chris Paxton and Jiafei Duan!

Teaching robots from human video is an important part of overcoming the “data gap” in robotics, but many of the details still need to be worked out. Homanga Bharadhwaj tells us about two recent research papers, Gen2Act and Spider, which go over different aspects of the problem: Gen2Act uses generative video models to create a reference of how a task should be performed given a language prompt; then, it uses a multi-purpose policy that can “translate” from human video to robot motion. However, Gen2Act has its limitations, in particular when it comes to dexterous, contact-rich tasks. That’s where SPIDER comes in: it uses human data together with simulation to train policies across many different humanoid hands and datasets. Also of note is that this is our first episode with our new rotating co-host, Jiafei Duan. To learn more, watch Episode #57 of RoboPapers now, with Chris Paxton and Jiafei Duan!

RoboPapers

24,286 Aufrufe • vor 8 Monaten

Robotics has a data problem, and egocentric data is a compelling way to solve it because egocentric data is (1) scalable — it’s cheap to collect — and (2) easily captures the breadth and diversity of real human tasks, while (3) being true to real physics. Lightwheel recently released a hundred thousand hours of high-quality egocentric video data, spread across a wide range of human tasks from cleaning and tidying to assembly and construction to retail tasks. Here to tell us about this tremendous effort is Steve Xie, CEO and co-founder of Lightwheel. The data is also available on HuggingFace right now. Watch Episode 102 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, to learn more!

Robotics has a data problem, and egocentric data is a compelling way to solve it because egocentric data is (1) scalable — it’s cheap to collect — and (2) easily captures the breadth and diversity of real human tasks, while (3) being true to real physics. Lightwheel recently released a hundred thousand hours of high-quality egocentric video data, spread across a wide range of human tasks from cleaning and tidying to assembly and construction to retail tasks. Here to tell us about this tremendous effort is Steve Xie, CEO and co-founder of Lightwheel. The data is also available on HuggingFace right now. Watch Episode 102 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, to learn more!

RoboPapers

33,470 Aufrufe • vor 19 Tagen

Human skin plays an important role in how we interact with the world and robustly manipulate objects. It’s not just important when we can’t see things with out eyes, but when we want to pick up something heavy, or apply a very specific amount of force. So, it makes sense to want to give robots skin. Enter DexSkin: a soft, deformable electronic skin which can be applied across different surfaces and used to cover robot hands or fingers. Suzannah Wistreich and Baiyu Shi talk to us about their work building DexSkin, showing how it’s useful for policy learning, including online reinforcement learning, and how it' can be calibrated and policies transferred across sensors. They also open sourced their code and methods for building the sensors. To learn more, watch Episode #88 of RoboPapers now, hosted by Chris Paxton and Jiafei Duan!

Human skin plays an important role in how we interact with the world and robustly manipulate objects. It’s not just important when we can’t see things with out eyes, but when we want to pick up something heavy, or apply a very specific amount of force. So, it makes sense to want to give robots skin. Enter DexSkin: a soft, deformable electronic skin which can be applied across different surfaces and used to cover robot hands or fingers. Suzannah Wistreich and Baiyu Shi talk to us about their work building DexSkin, showing how it’s useful for policy learning, including online reinforcement learning, and how it' can be calibrated and policies transferred across sensors. They also open sourced their code and methods for building the sensors. To learn more, watch Episode #88 of RoboPapers now, hosted by Chris Paxton and Jiafei Duan!

RoboPapers

21,053 Aufrufe • vor 2 Monaten

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

RoboPapers

19,357 Aufrufe • vor 5 Monaten

Diffusion Policy was one of the big breakthroughs that has enabled an explosion in real-world robot learning. However, it’s always had a weakness, which is that it works by computing a final action trajectory from random noise, which leads to high latency when predicting a final action sequence. Instead, why not initialize the search based on previous actions? This allows for incredibly fast policy inference and in many cases improved generalization, generating high-quality predictions with sub-ms latency. Jindou Jia and Jianfei Yang join us to explain. Learn more on Episode 95 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton!

Diffusion Policy was one of the big breakthroughs that has enabled an explosion in real-world robot learning. However, it’s always had a weakness, which is that it works by computing a final action trajectory from random noise, which leads to high latency when predicting a final action sequence. Instead, why not initialize the search based on previous actions? This allows for incredibly fast policy inference and in many cases improved generalization, generating high-quality predictions with sub-ms latency. Jindou Jia and Jianfei Yang join us to explain. Learn more on Episode 95 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

28,528 Aufrufe • vor 1 Monat

Spatial understanding is important to moving around in complex environments and is a huge part of the challenge of generalizing to new scenes. Most world models, however, largely ignore this spatial dimension, focusing on 2D images. Not PointWorld, though. PointWorld is a 3D world model trained from real and simulated data which can perform a wide variety of manipulation tasks on a real robot, including grasping or handling articulated objects, all without any additional fine tuning. Wenlong Huang joins us to tell us more about what makes this work and how it’s different from other world models. Watch Episode #83 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Spatial understanding is important to moving around in complex environments and is a huge part of the challenge of generalizing to new scenes. Most world models, however, largely ignore this spatial dimension, focusing on 2D images. Not PointWorld, though. PointWorld is a 3D world model trained from real and simulated data which can perform a wide variety of manipulation tasks on a real robot, including grasping or handling articulated objects, all without any additional fine tuning. Wenlong Huang joins us to tell us more about what makes this work and how it’s different from other world models. Watch Episode #83 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

16,197 Aufrufe • vor 4 Monaten