Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Teleoperating a robot is hard. This means that when performing a robot task via teleoperation — say, to collect examples for training a robot policy — it’s almost unavoidably slower than you would like, below either the capabilities of the human expert on their own or the robot performing... the task. Wouldn’t it be great if there was a way to fix this? Unfortunately, it’s harder than it looks. You can’t just execute faster, as this alters the distribution of environment states the policy will encounter. Nadun Ranawaka Arachchige and Zhenyang Chen propose Speed-Adaptive Imitation Learning (SAIL), which adds error-adaptive guidance, adapts execution speed according to task structure, predicts controller-invariant action targets to ensure robustness across execution speeds, and explicitly models delays from, for example, sensor latency. Watch episode #59 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton to learn more!show more

RoboPapers

3,966 subscribers

27,970 views • 6 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

18,813 views • 7 months ago

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

RoboPapers

27,344 views • 3 months ago

We’ve all seen videos of humanoid robots performing single tasks that are very impressive, like dancing or karate. But training humanoid robots to perform a wide range of complex motions is difficult. GMT is a general-purpose policy which can learn a wide range of robot motions. Watch Episode #32 of RoboPapers, with Zixuan Chen , co-hosted by Michael Cho - Rbt/Acc and Chris Paxton , to learn more.

We’ve all seen videos of humanoid robots performing single tasks that are very impressive, like dancing or karate. But training humanoid robots to perform a wide range of complex motions is difficult. GMT is a general-purpose policy which can learn a wide range of robot motions. Watch Episode #32 of RoboPapers, with Zixuan Chen , co-hosted by Michael Cho - Rbt/Acc and Chris Paxton , to learn more.

RoboPapers

16,605 views • 10 months ago

Collecting dexterous humanoid robot data is difficult to scale. That's why Mengda Xu and Han Zhang built DexUMI: a tool for demonstrating how to control a dexterous robot hand, which allows you to quickly collect task data. Co-hosted by Michael Cho - Rbt/Acc and Chris Paxton

Collecting dexterous humanoid robot data is difficult to scale. That's why Mengda Xu and Han Zhang built DexUMI: a tool for demonstrating how to control a dexterous robot hand, which allows you to quickly collect task data. Co-hosted by Michael Cho - Rbt/Acc and Chris Paxton

RoboPapers

26,344 views • 1 year ago

Evaluating robot policies is hard. Every lab has a different robot; reproducible evaluations are really challenging. This makes it hard for us to know which methods for learning robot policies are likely to perform the best in real-world scenarios. Taking a page from LLM evaluations like Chatbot Arena, RoboArena aims to address this problem through crowdsourcing evaluations with a network of different evaluators. Watch Episode #34 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton , now to learn more from authors Pranav Atreya and Karl Pertsch!

Evaluating robot policies is hard. Every lab has a different robot; reproducible evaluations are really challenging. This makes it hard for us to know which methods for learning robot policies are likely to perform the best in real-world scenarios. Taking a page from LLM evaluations like Chatbot Arena, RoboArena aims to address this problem through crowdsourcing evaluations with a network of different evaluators. Watch Episode #34 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton , now to learn more from authors Pranav Atreya and Karl Pertsch!

RoboPapers

15,056 views • 10 months ago

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

RoboPapers

89,011 views • 9 months ago

Most robot policies today still largely lack memory: they make all their decisions based on what they can see right now. MemER aims to change that by learning which frames are important; this lets it deal with tasks like object search. Ajay Sridhar, Jenny Pan, and @satviks107Sharma tell us about how to achieve this fundamental capability for long-horizon task execution. Watch Episode #54 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

Most robot policies today still largely lack memory: they make all their decisions based on what they can see right now. MemER aims to change that by learning which frames are important; this lets it deal with tasks like object search. Ajay Sridhar, Jenny Pan, and @satviks107Sharma tell us about how to achieve this fundamental capability for long-horizon task execution. Watch Episode #54 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

RoboPapers

17,850 views • 7 months ago

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

27,567 views • 6 months ago

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

20,434 views • 7 months ago

For robots to be useful, they can’t just dance — they must be able to physically interact with the world around them. Unfortunately, the sorts of motion tracking policies you see performing dancing or martial arts are not really capable of the kind of precise, forceful interaction needed to perform useful interactions with the world. Siheng Zhao and Yanjie Ze join us to talk about ResMimic, their new paper which takes a general-purpose human motion-tracking policy and improves it with a residual policy to reliably interact with objects To learn more, watch Episode #47 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, today!

For robots to be useful, they can’t just dance — they must be able to physically interact with the world around them. Unfortunately, the sorts of motion tracking policies you see performing dancing or martial arts are not really capable of the kind of precise, forceful interaction needed to perform useful interactions with the world. Siheng Zhao and Yanjie Ze join us to talk about ResMimic, their new paper which takes a general-purpose human motion-tracking policy and improves it with a residual policy to reliably interact with objects To learn more, watch Episode #47 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, today!

RoboPapers

11,399 views • 8 months ago

$We’re excited to launch Instance – a success detector for robot rollouts! Describe your task and drop in a dataset, and it judges success/fail on each rollout with detailed subtask captions– it’s more accurate than Claude Opus 4.8, at a fraction of the latency. This is just the first step toward our larger goal: an autonomous evaluation rig for robot learning. Today, evaluating a robot policy means humans have to watch the robot roll out, mark success, and reset the scene– we're automating that, starting with the success judge.$

We’re excited to launch Instance – a success detector for robot rollouts! Describe your task and drop in a dataset, and it judges success/fail on each rollout with detailed subtask captions– it’s more accurate than Claude Opus 4.8, at a fraction of the latency. This is just the first step toward our larger goal: an autonomous evaluation rig for robot learning. Today, evaluating a robot policy means humans have to watch the robot roll out, mark success, and reset the scene– we're automating that, starting with the success judge.

Lucy Cai

72,408 views • 1 month ago

How can we build robotic hands with truly superhuman dexterity? Daxo Robotics is developing a unique tendon-driven soft robot hand, which aims to be tougher and more capable than a traditional humanoid hand. Each finger consists of many different tendons, which act in concert to move or manipulate. This is a special episode of RoboPapers where, instead of talking about a scientific paper, we talk to Tom Zhang, founder of Daxo Robotics, in order to learn both about his background and about how this one-of-a-kind robot hand design works. Watch Episode #40 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now!

How can we build robotic hands with truly superhuman dexterity? Daxo Robotics is developing a unique tendon-driven soft robot hand, which aims to be tougher and more capable than a traditional humanoid hand. Each finger consists of many different tendons, which act in concert to move or manipulate. This is a special episode of RoboPapers where, instead of talking about a scientific paper, we talk to Tom Zhang, founder of Daxo Robotics, in order to learn both about his background and about how this one-of-a-kind robot hand design works. Watch Episode #40 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

16,321 views • 9 months ago

How can we build a general-purpose “foundation model” for robot motion? Zhengyi “Zen” Luo joins us to talk about SONIC, which uses motion tracking as a foundational task for humanoid robot control, and scales humanoid control training to 9k GPU hours and 100 million frames worth of data. The result: a model with a generally-useful embedding space that can be controlled by a VLA, or from human video, to perform a wide variety of humanoid whole-body-control tasks, including with zero-shot transfer to previously unseen motions. Watch episode 72 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

How can we build a general-purpose “foundation model” for robot motion? Zhengyi “Zen” Luo joins us to talk about SONIC, which uses motion tracking as a foundational task for humanoid robot control, and scales humanoid control training to 9k GPU hours and 100 million frames worth of data. The result: a model with a generally-useful embedding space that can be controlled by a VLA, or from human video, to perform a wide variety of humanoid whole-body-control tasks, including with zero-shot transfer to previously unseen motions. Watch episode 72 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan, now!

RoboPapers

24,802 views • 3 months ago

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,334 views • 2 months ago

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!

RoboPapers

30,004 views • 3 months ago

Training robot foundation models faces two key hurdles: how to get enough data to train an effective model, and how to make sure that new skills can be acquired quickly. The team at Rhoda AI believes that the answer is training Direct Video Action models from web data. Web data is plentiful, to the point where Rhoda can train their base model on hundreds of years of video data. And then, with the addition of robot data, they can quickly adapt it to new tasks with as little as 20 hours of in-domain data, performing complex, multi-step manipulation tasks with their purpose-built video foundation model. Tongzhou Mu 🤖🦾🦿 Eric Chan and Changan Chen joined us to talk more about their approach. Watch Episode #79 of RoboPapers, with Michael Cho - Rbt/Acc, Chris Paxton, and Jiafei Duan, to learn more!

Training robot foundation models faces two key hurdles: how to get enough data to train an effective model, and how to make sure that new skills can be acquired quickly. The team at Rhoda AI believes that the answer is training Direct Video Action models from web data. Web data is plentiful, to the point where Rhoda can train their base model on hundreds of years of video data. And then, with the addition of robot data, they can quickly adapt it to new tasks with as little as 20 hours of in-domain data, performing complex, multi-step manipulation tasks with their purpose-built video foundation model. Tongzhou Mu 🤖🦾🦿 Eric Chan and Changan Chen joined us to talk more about their approach. Watch Episode #79 of RoboPapers, with Michael Cho - Rbt/Acc, Chris Paxton, and Jiafei Duan, to learn more!

RoboPapers

24,475 views • 3 months ago

Robotics has changed dramatically over the last eight years. Ted Xiao has been involved in the cutting edge of robot learning through this period, spending those eight years at Google Brain/Google Deepmind. And he’s identified three eras of robot learning. These eras are: - The Era of Existence Proofs - trying different methods like QT-Opt, on-robot RL - The Era of Foundation Models - transitioning to data collection and clean objectives (i.e. supervised learning) - The Era of Scaling - orders of magnitude more data and larger models, enabling reasoning, long-horizon actions, and cross-embodiment transfer Watch Episode 78 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan to learn more!

Robotics has changed dramatically over the last eight years. Ted Xiao has been involved in the cutting edge of robot learning through this period, spending those eight years at Google Brain/Google Deepmind. And he’s identified three eras of robot learning. These eras are: - The Era of Existence Proofs - trying different methods like QT-Opt, on-robot RL - The Era of Foundation Models - transitioning to data collection and clean objectives (i.e. supervised learning) - The Era of Scaling - orders of magnitude more data and larger models, enabling reasoning, long-horizon actions, and cross-embodiment transfer Watch Episode 78 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan to learn more!

RoboPapers

36,732 views • 3 months ago

The problem with humanoid teleoperation is that it is expensive and difficult to scale Enter NVIDIA's EgoScale: - A VLA model pretrained on thousands hours of egocentric human videos. - Mid-trained via 50 hours of human + 4 hours of robot "play" data for human-robot alignment. - Fine-tuned with very few examples of task-specific robot teleoperation (100 or fewer per task). - Successfully transfers across 5-finger (Sharpa) and 3-finger (Unitree G1) robot hands. - Performance scales predictably as data increases.

The problem with humanoid teleoperation is that it is expensive and difficult to scale Enter NVIDIA's EgoScale: - A VLA model pretrained on thousands hours of egocentric human videos. - Mid-trained via 50 hours of human + 4 hours of robot "play" data for human-robot alignment. - Fine-tuned with very few examples of task-specific robot teleoperation (100 or fewer per task). - Successfully transfers across 5-finger (Sharpa) and 3-finger (Unitree G1) robot hands. - Performance scales predictably as data increases.

The Humanoid Hub

44,441 views • 5 months ago

Learning policies via imitation is extremely potent, but making sure those policies will generalize to out of distribution settings is still very hard. SAILOR proposes a solution in learning to search via a learned world model, which outperforms existing imitation approaches. Gokul Swamy, Arnav Jain , and Vibhakar Mohta tell us about their approach. Watch Episode #33 of RoboPapers, co-hosted by Michael Cho - Rbt/Acc and Chris Paxton , now!

Learning policies via imitation is extremely potent, but making sure those policies will generalize to out of distribution settings is still very hard. SAILOR proposes a solution in learning to search via a learned world model, which outperforms existing imitation approaches. Gokul Swamy, Arnav Jain , and Vibhakar Mohta tell us about their approach. Watch Episode #33 of RoboPapers, co-hosted by Michael Cho - Rbt/Acc and Chris Paxton , now!

RoboPapers

21,440 views • 10 months ago

Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video. Elvis Nava joins us to talk about @mimicrobotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM. Watch Episode #81 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

Robotics fundamentally involves understanding the dynamics of how things change in the world in response to action and force. This is impossible to learn from static images; instead, it’s far more effective and more data-efficient to learn from video. Elvis Nava joins us to talk about @mimicrobotic. One of the key findings from mimic-video is that pretraining on webscale video allows robots to learn physics priors; as a result, policies train faster, generalize better, and are capable of more impressive dexterity, versus training on static images or image-language pairs as per a VLM. Watch Episode #81 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton to learn more!

RoboPapers

48,343 views • 2 months ago