Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Can we scale robot learning without scaling human labor? 📈 Demonstrations, resets, failure recovery, constant supervision—collecting robot data still takes substantial human effort. 🔑 Three keys let agents take on more of the work: 1. Real2Sim2Real: Agents explore and learn in digital twins, where trial and error is safer...

Isabella__Liu's profile picture

Isabella Liu

1,078 subscribers

16,228 Aufrufe • vor 12 Tagen •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Isabella Liu
Isabella Liuvor 12 Tagen

1/ Real2Sim2Real 🌎 Real-world exploration is slow, costly, and sometimes dangerous. With Recova, a coding agent reconstructs a real robot workstation as a digital twin, then explores tasks, failures, and recovery strategies in simulation. Successful trials become training data, while failures guide the discovery of new recovery skills. The agent handles the trial and error in simulation, then brings the learned skills back to the real world.

Profilbild von Isabella Liu
Isabella Liuvor 12 Tagen

2/ Autonomous Recovery 🔄 Recova trains a separate recovery policy alongside the task policy, conditioned on recovery instructions. During real-world rollouts, a vision-language monitor watches the robot. When the task policy fails, the monitor identifies the failure, calls a matching recovery skill, checks that the scene is restored, and resumes the task.

Profilbild von Isabella Liu
Isabella Liuvor 12 Tagen

3/ Human-on-Demand Learning 🙋 Recova can run several robot stations in parallel, each watched by a vision-language monitor that handles failures with autonomous recovery. A human steps in only when autonomous recovery falls short. Their demonstration becomes training data for the next round, so similar failures can later be handled without them.

Profilbild von Isabella Liu
Isabella Liuvor 12 Tagen

4/ Simulation Benchmarks 📊 Do recovery skills hold up in simulation? We tested on two public benchmarks. On LIBERO-Pro, Recova reaches 78.8% mean success vs. 71.7% for the best published method. On MolmoSpaces, it reaches 64.9% vs. 38.0%.

Profilbild von Isabella Liu
Isabella Liuvor 12 Tagen

5/ Recova plays mahjong 🀄😃 The agent picks each move, the arm plays it, and recovery skills step in when something goes wrong.

Profilbild von Isabella Liu
Isabella Liuvor 12 Tagen

Huge thanks to my collaborators, and to Cristaldo Campos, Jimmy Wu, and Tingwu Wang for their help with the robot setup and demo filming. 📄 🌐

Profilbild von Nana | Japan Physical AI Research
Nana | Japan Physical AI Researchvor 12 Tagen

The scaling bottleneck may be less “collect more data” and more “remove human operations from the collection loop.” I’d watch whether resets, recovery and supervision per useful episode actually fall as the system improves.

Profilbild von Keyu Zhu
Keyu Zhuvor 11 Tagen

great work!

Profilbild von Taehyeon Kim
Taehyeon Kimvor 12 Tagen

How does the system decide autonomous recovery has failed and a human should step in? A learned success detector, or timeouts and progress thresholds?

Profilbild von clankr
clankrvor 12 Tagen

So this is like distillation but for robotics right? Cool work

Ähnliche Videos

LimX Dynamics demonstrated autonomous unboxing with its humanoid robot Oli, requiring almost no human effort. The robot can step out of its crate and become fully operational on its own without manual setup. Multiple robots can be activated together and controlled through a single system, saving time and effort.
0:28

LimX Dynamics demonstrated autonomous unboxing with its humanoid robot Oli, requiring almost no human effort. The robot can step out of its crate and become fully operational on its own without manual setup. Multiple robots can be activated together and controlled through a single system, saving time and effort.

Space and Technology

31,963 Aufrufe • vor 6 Monaten

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n
1:28

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

Zhi (Leo) Wang

128,190 Aufrufe • vor 5 Monaten

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵
0:55

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵

Mahi Shafiullah 🏠🤖

97,144 Aufrufe • vor 4 Monaten

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗
4:41

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

Chenhao Li

22,062 Aufrufe • vor 4 Monaten

A million hours of what? If we’re scaling human demonstrations for robot learning, what should each hour contain? We built S1, our skill capture glove, alongside our robot hand to record joint movements that map to robot targets, with touch and video on the same timeline.
1:56

A million hours of what? If we’re scaling human demonstrations for robot learning, what should each hour contain? We built S1, our skill capture glove, alongside our robot hand to record joint movements that map to robot targets, with touch and video on the same timeline.

Quanting Xie

49,781 Aufrufe • vor 4 Tagen

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇
1:06

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇

Tony Tao

97,762 Aufrufe • vor 1 Jahr

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian ✈️ IROS join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!
55:30

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian ✈️ IROS join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!

RoboPapers

30,095 Aufrufe • vor 5 Monaten

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.
2:02

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

Physical Intelligence

1,188,956 Aufrufe • vor 10 Monaten

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:
0:47

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

22,891 Aufrufe • vor 11 Monaten

Can VLMs enable robots to autonomously improve? In our new work we ran a fleet of robot arms to collect autonomous data with VLM-proposed tasks and showed that robots can keep getting better as they are deployed, without supervision: 🧵👇
1:37

Can VLMs enable robots to autonomously improve? In our new work we ran a fleet of robot arms to collect autonomous data with VLM-proposed tasks and showed that robots can keep getting better as they are deployed, without supervision: 🧵👇

Sergey Levine

19,449 Aufrufe • vor 2 Jahren

Experiments in progress. The one on the right has been learning for ~3 hours, the one in the middle for ~1 hour, and the one on the left just started a few minutes ago. The initial motivation for making the physical Atari was just to commit ourselves to a subset of algorithms that can make progress in this setup. This commitment rules out algorithms that require billions of samples to learn (or worse, require multiple environments running in parallel). Atari games are simple enough that we should be able to show learning on them in a short amount of time with no prior knowledge. Since then, I've realized that this setup is also a good way to compare different paradigms in robotics in a principled way. These paradigms are sim2real, learning from tele-operated data, and learning directly on the robots. So far, I have observed that getting sim2real to work reliably is hard. It requires tweaks that don't scale. Policies that can play perfectly in simulation fall apart because of latencies and the messiness of the real world. These aspects could be modeled to improve the simulation, but not without sinking significant human engineering hours. I have higher hopes for learning from tele-operated data, but that requires a human to learn the task first. These experiments are on my to-do list. I have to learn to play some of the games well through the robot. I’m half-decent at playing Pong and Ms Pacman now. Learning directly on robots is looking like the most promising approach. This approach takes away pesky distribution shifts and makes it possible to have algorithms that continually improve with more data and time without any human intervention. It feels great to let experiments run overnight and wake up to find improved policies. With learning on robots, I should, in principle, be able to go on a long vacation and come back to find better policies for complex tasks beyond Atari games. Whether that is possible with current learning algorithms is a different question.
1:05

Experiments in progress. The one on the right has been learning for ~3 hours, the one in the middle for ~1 hour, and the one on the left just started a few minutes ago. The initial motivation for making the physical Atari was just to commit ourselves to a subset of algorithms that can make progress in this setup. This commitment rules out algorithms that require billions of samples to learn (or worse, require multiple environments running in parallel). Atari games are simple enough that we should be able to show learning on them in a short amount of time with no prior knowledge. Since then, I've realized that this setup is also a good way to compare different paradigms in robotics in a principled way. These paradigms are sim2real, learning from tele-operated data, and learning directly on the robots. So far, I have observed that getting sim2real to work reliably is hard. It requires tweaks that don't scale. Policies that can play perfectly in simulation fall apart because of latencies and the messiness of the real world. These aspects could be modeled to improve the simulation, but not without sinking significant human engineering hours. I have higher hopes for learning from tele-operated data, but that requires a human to learn the task first. These experiments are on my to-do list. I have to learn to play some of the games well through the robot. I’m half-decent at playing Pong and Ms Pacman now. Learning directly on robots is looking like the most promising approach. This approach takes away pesky distribution shifts and makes it possible to have algorithms that continually improve with more data and time without any human intervention. It feels great to let experiments run overnight and wake up to find improved policies. With learning on robots, I should, in principle, be able to go on a long vacation and come back to find better policies for complex tasks beyond Atari games. Whether that is possible with current learning algorithms is a different question.

Khurram Javed

52,110 Aufrufe • vor 10 Monaten

Can robots self-improve by collecting data autonomously🤖? Introducing SOAR: a system for large-scale autonomous data collection 🚀 and autonomous improvement📈of a multi-task language-conditioned policy in diverse scenes without human interventions .
1:36

Can robots self-improve by collecting data autonomously🤖? Introducing SOAR: a system for large-scale autonomous data collection 🚀 and autonomous improvement📈of a multi-task language-conditioned policy in diverse scenes without human interventions .

Paul Zhou

47,739 Aufrufe • vor 2 Jahren

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]
1:48

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Irmak Guzey

65,141 Aufrufe • vor 2 Jahren

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄
0:32

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

23,562 Aufrufe • vor 3 Monaten

🇨🇳 It has started. A new home service in China pairs human cleaners with autonomous AI robots to tackle household chores. Residents in Shenzhen can now book a service where a human professional and an autonomous robot arrive together to clean their home. Real houses present a chaotic mess of dropped toys and random furniture that confuse traditional machines. X Square Robot and a major service platform named 58[.]com decided to tackle this chaos by launching China's first robot cleaner service in March-26. Customers use an application to hire a cleaning crew that consists of 1 human worker and 1 robot. The human takes care of the tricky chores that require complex judgment. The robot handles the repetitive physical work like picking up trash and wiping down flat surfaces. This machine runs on a system called WALL-A, which acts as a single continuous AI brain rather than a list of pre-written rules. They built this AI foundation model to perceive its surroundings and make its own decisions without human guidance. It processes visual data and plans multi-step actions. And deploying these robots into actual homes now provides the massive amounts of extremely important training data to improve it continuously. Alibaba and ByteDance backed this project. IMO, if the foundational model behind it figures out how to navigate a messy living room without getting stuck, it can learn to operate in almost any other physical environment.
1:19

🇨🇳 It has started. A new home service in China pairs human cleaners with autonomous AI robots to tackle household chores. Residents in Shenzhen can now book a service where a human professional and an autonomous robot arrive together to clean their home. Real houses present a chaotic mess of dropped toys and random furniture that confuse traditional machines. X Square Robot and a major service platform named 58[.]com decided to tackle this chaos by launching China's first robot cleaner service in March-26. Customers use an application to hire a cleaning crew that consists of 1 human worker and 1 robot. The human takes care of the tricky chores that require complex judgment. The robot handles the repetitive physical work like picking up trash and wiping down flat surfaces. This machine runs on a system called WALL-A, which acts as a single continuous AI brain rather than a list of pre-written rules. They built this AI foundation model to perceive its surroundings and make its own decisions without human guidance. It processes visual data and plans multi-step actions. And deploying these robots into actual homes now provides the massive amounts of extremely important training data to improve it continuously. Alibaba and ByteDance backed this project. IMO, if the foundational model behind it figures out how to navigate a messy living room without getting stuck, it can learn to operate in almost any other physical environment.

Rohan Paul

53,039 Aufrufe • vor 7 Monaten

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video
0:24

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 Aufrufe • vor 1 Jahr

The most frustrating part of imitation learning is collecting huge amounts of teleop data. But why teleop robots when robots can learn by watching us? Introducing Point Policy, a novel framework that enables robots to learn from human videos without any teleop, sim2real, or RL.
0:40

The most frustrating part of imitation learning is collecting huge amounts of teleop data. But why teleop robots when robots can learn by watching us? Introducing Point Policy, a novel framework that enables robots to learn from human videos without any teleop, sim2real, or RL.

Siddhant Haldar

69,056 Aufrufe • vor 1 Jahr

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:
0:29

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 Aufrufe • vor 8 Monaten