正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Lerrel Pinto

9,977 subscribers

42,673 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

11 条评论

Lerrel Pinto 的头像

Lerrel Pinto1 年前

EgoZero trains on in-the-wild egocentric data (~20 minutes of human FPV data). In all experiments, we collect no data from our test-time environment to emphasize the zero-shot capability of our models.

Lerrel Pinto 的头像

Lerrel Pinto1 年前

The key technical idea in EgoZero is to represent the scene as 3D points. This allows for: (a) Easy transfer of latent representation from human to robot. (b) Larger spatial (translation & rotation) generalization as we can easily data-augment for this.

Lerrel Pinto 的头像

Lerrel Pinto1 年前

An immediate byproduct of EgoZero’s 3D point-based policies is that the robot performance is fairly robust to camera viewpoints.

Lerrel Pinto 的头像

Lerrel Pinto1 年前

EgoZero builds on top of the Point Policy architecture from @haldar_siddhant -- these point representations are a game changer. Our work is also inspired by excellent work on co-training egocentric data and robot teleop data from @danfei_xu and @xiaolonw among others.

Lerrel Pinto 的头像

Lerrel Pinto1 年前

We still have several open challenges here that we are actively working on: (a) Addressing errors caused by human to robot correspondences. (b) Improving pose tracking without relying on stationary keypoints for SfM. (c) Improving upon the 1-2cm error in hand pose estimation.

Lerrel Pinto 的头像

Lerrel Pinto1 年前

For more details, paper and videos, check out our website: This work was led by @vincentjliu @AdemiAdeniji & @haotianz_David and a wonderful collaboration with @Raunaqmb & @pabbeel

Lerrel Pinto 的头像

Lerrel Pinto1 年前

@AdemiAdeniji @haotianz_David @Raunaqmb @pabbeel Here is the website without the typo:

NICE 的头像

NICE1 年前

Stay competitive by balancing cutting-edge AI with automation tools. Forrester shows how.

Aman Goyal 的头像

Aman Goyal1 年前

Looks really exciting. Looking forward to your next paper.

Fadi "Ali" 的头像

Fadi "Ali"1 年前

EgoZero's a real "future-tech" step! Robots training with human data is sci-fi gold. Here's to reshaping our tech landscape and pushing boundaries. 🚀

cookies (🍪,🍪) | 饼妹的头像

cookies (🍪,🍪) | 饼妹1 年前

Will the new skills learnt only be applicable to certain use cases, instead of catering to general scenarios?

相关视频

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇

Tony Tao

97,762 次观看 • 1 年前

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Lerrel Pinto

56,591 次观看 • 2 年前

Introducing Mobi-π: Mobilizing Your Robot Learning Policy. Our method: ✈️ enables flexible mobile skill chaining 🪶 without requiring additional policy training data 🏠 while scaling to unseen scenes 🧵↓

Introducing Mobi-π: Mobilizing Your Robot Learning Policy. Our method: ✈️ enables flexible mobile skill chaining 🪶 without requiring additional policy training data 🏠 while scaling to unseen scenes 🧵↓

Jingyun Yang

59,194 次观看 • 1 年前

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes even*cheaper* with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes evencheaper with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

Abhishek Gupta

13,358 次观看 • 1 年前

Can VLMs enable robots to autonomously improve? In our new work we ran a fleet of robot arms to collect autonomous data with VLM-proposed tasks and showed that robots can keep getting better as they are deployed, without supervision: 🧵👇

Can VLMs enable robots to autonomously improve? In our new work we ran a fleet of robot arms to collect autonomous data with VLM-proposed tasks and showed that robots can keep getting better as they are deployed, without supervision: 🧵👇

Sergey Levine

19,449 次观看 • 2 年前

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲

Yang Gao

125,462 次观看 • 1 年前

Excited to release FAST, our new robot action tokenizer! 🤖 Some highlights: - Simple autoregressive VLAs match diffusion VLA performance - Trains up to 5x faster - Works on all robot datasets we tested - First VLAs that work out-of-the-box in new environments! 🧵/

Excited to release FAST, our new robot action tokenizer! 🤖 Some highlights: - Simple autoregressive VLAs match diffusion VLA performance - Trains up to 5x faster - Works on all robot datasets we tested - First VLAs that work out-of-the-box in new environments! 🧵/

Karl Pertsch

90,725 次观看 • 1 年前

Robot-free demos can be collected through our VR interface, inspected, trained, and evaluated in a closed loop. Cool thing is that a small amount of real-robot data mixed with large-scale robot-free data can reach comparable performance, while reducing real-robot data needs by up to 20x. Code: Paper:

Robot-free demos can be collected through our VR interface, inspected, trained, and evaluated in a closed loop. Cool thing is that a small amount of real-robot data mixed with large-scale robot-free data can reach comparable performance, while reducing real-robot data needs by up to 20x. Code: Paper:

X Square Robot

175,588 次观看 • 3 个月前

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 次观看 • 1 年前

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

Zhi (Leo) Wang

111,277 次观看 • 3 个月前

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 次观看 • 1 年前

Collect robot demos from anywhere through AR! Excited to introduce 🎯DART, Dexterous AR Teleoperation interface enabling anyone to teleoperate robots in cloud-hosted simulation. With DART, anyone can collect robot demos anywhere, anytime, for multiple robots and tasks in one sitting. Every data is automatically logged on our open-sourced cloud database DexHub for public use. 🧵[1/n]

Collect robot demos from anywhere through AR! Excited to introduce 🎯DART, Dexterous AR Teleoperation interface enabling anyone to teleoperate robots in cloud-hosted simulation. With DART, anyone can collect robot demos anywhere, anytime, for multiple robots and tasks in one sitting. Every data is automatically logged on our open-sourced cloud database DexHub for public use. 🧵[1/n]

Younghyo Park

27,464 次观看 • 1 年前

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 次观看 • 1 年前

Imitation learning works™ – but you need good data 🥹 How to get high-quality visuotactile demos from a bimanual robot with multifingered hands, and learn smooth policies? Check our new work “Learning Visuotactile Skills with Two Multifingered Hands”! 🙌

Imitation learning works™ – but you need good data 🥹 How to get high-quality visuotactile demos from a bimanual robot with multifingered hands, and learn smooth policies? Check our new work “Learning Visuotactile Skills with Two Multifingered Hands”! 🙌

Toru

61,664 次观看 • 2 年前

Chelsea Finn (Chelsea Finn) on building general-purpose robotics, and bringing intelligence into the physical world. At AI Startup School in San Francisco. 00:00 - General Purpose Robots 00:11 - Challenges in Robotics Applications 00:57 - Physical Intelligence: A New Approach 01:47 - Learning from Language Models 02:08 - Data Sources for Training Robots 03:32 - Training with Real-World Data 04:39 - Initial Successes and Challenges 09:10 - Breakthrough in Robot Training 11:03 - Improving Performance 15:43 - Expanding Capabilities 17:34 - Robots in Unseen Environments 25:54 - Handling Open-Ended Prompts 29:36 - Evaluating Robot Performance 30:03 - Future Directions and Challenges 31:27 - Audience Q&A

Chelsea Finn (Chelsea Finn) on building general-purpose robotics, and bringing intelligence into the physical world. At AI Startup School in San Francisco. 00:00 - General Purpose Robots 00:11 - Challenges in Robotics Applications 00:57 - Physical Intelligence: A New Approach 01:47 - Learning from Language Models 02:08 - Data Sources for Training Robots 03:32 - Training with Real-World Data 04:39 - Initial Successes and Challenges 09:10 - Breakthrough in Robot Training 11:03 - Improving Performance 15:43 - Expanding Capabilities 17:34 - Robots in Unseen Environments 25:54 - Handling Open-Ended Prompts 29:36 - Evaluating Robot Performance 30:03 - Future Directions and Challenges 31:27 - Audience Q&A

Y Combinator

129,834 次观看 • 1 年前

We got a robot to clean up homes that were never seen in its training data! Our new model, π-0.5, aims to tackle open-world generalization. We took our robot into homes that were not in the training data and asked it to clean kitchens and bedrooms. More below⤵️

We got a robot to clean up homes that were never seen in its training data! Our new model, π-0.5, aims to tackle open-world generalization. We took our robot into homes that were not in the training data and asked it to clean kitchens and bedrooms. More below⤵️

Physical Intelligence

535,573 次观看 • 1 年前

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Soroush Nasiriany

27,495 次观看 • 1 年前

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,466 次观看 • 2 年前

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

Chenhao Li

22,062 次观看 • 3 个月前