正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

🤖 Robotics often faces a chicken and egg problem: no web-scale robot data for training (unlike CV or NLP) b/c robots aren't deployed yet & vice-versa. Introducing VRB: Use large-scale human videos to train a general-purpose affordance model to jumpstart any robotics paradigm!

Deepak Pathak

30,944 subscribers

70,743 次观看 • 3 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

10 条评论

Deepak Pathak 的头像

Deepak Pathak3 年前

🌐 Given a new scene, our general-purpose VRB model predicts all the locations *where* a robot can manipulate objects and *how* should it move post-manipulation. 💡Qs: 1) What are affordances? 2) How to get large-scale data? 3) How does this help robots?

Deepak Pathak 的头像

Deepak Pathak3 年前

🧩 In computer vision, there is a long line of work in affordances (Gibson 1966, 1979). However, what's the best way to define them for robotics? Our solution: interaction points & post-contact trajectories, allowing for seamless integration with almost any robot learning setup.

Deepak Pathak 的头像

Deepak Pathak3 年前

🔧🔧 How do we extract affordances from humans? We estimate hand poses & hand-object interaction points to extract contact pts + interaction direction (thanks to @DandanShan_, David Fouhey's 100DoH), and map them back to frames where humans arent present to avoid embodiment gap.

Deepak Pathak 的头像

Deepak Pathak3 年前

How to use this model for different robot learning paradigms? We show 4 different paradigms. 1. Offline Data Collection 📊 Instead of using teleoperation or scripted policies, VRB can allow for an automatic collection of good quality interaction-rich data for policy learning.

Deepak Pathak 的头像

Deepak Pathak3 年前

2. Bootstrapping Exploration 🔍 Robots can utilize the predicted affordances from VRB to explore their environment more intelligently, leading to the discovery of novel and effective ways to manipulate objects.

Deepak Pathak 的头像

Deepak Pathak3 年前

3. Goal-Conditioned Learning 🎯 Can we use our affordance model to iteratively improve performance? We train goal-oriented policies using actions sampled from VRB. We prune this action distribution using the distance to the goal during the training process (using WHIRL RSS'22).

Deepak Pathak 的头像

Deepak Pathak3 年前

4. Action Space Reparameterization 🔄 We can also treat our affordances as an action space itself. It helps accelerate online RL eliminating the need for task-specific primitives. This enables DQN to learn manipulation tasks in under 30 mins!

Deepak Pathak 的头像

Deepak Pathak3 年前

Using internet-scale videos is a promising approach to tackling the data problem in robotics and we hope VRB is a strong step toward that! Led by @shikharbahl, @mendonca_rl, @lchen915, @unnatjain2010 CVPR 2023 Paper: Website: 9/9

Murtaza Dalal 的头像

Murtaza Dalal3 年前

Awesome work by my colleagues @shikharbahl, @mendonca_rl, @lchen915! Extracting affordances from human video enables robots to efficiently solve a wide array of manipulation tasks in the real world 🤖. Excited to see where this goes next!

Siyuan HUANG 的头像

Siyuan HUANG3 年前

Great ideas and a very interesting presentation. :-) I have checked your website but find the dataset is coming soon, would that be available soon?

相关视频

Introducing The Largest Organized and Incentivized Human-Data Fleet for General-Purpose Humanoid-Robotics and large language model training Our users are creators that complete daily data quests with crypto incentives through our Mobile App 🧵

Introducing The Largest Organized and Incentivized Human-Data Fleet for General-Purpose Humanoid-Robotics and large language model training Our users are creators that complete daily data quests with crypto incentives through our Mobile App 🧵

Mecka

386,000 次观看 • 2 年前

Can robot foundation models scale? Xiaomi-Robotics-1 explores this question with over 100,000 hours of real-world manipulation data. Pre-trained on large-scale real-world trajectories and post-trained with cross-embodiment robot data, Xiaomi-Robotics-1 demonstrates consistent scaling across data and model size, strong generalization in unseen environments, and efficient adaptation to new tasks. 🔗 #Robotics #EmbodiedAI #FoundationModels #RobotLearning #XiaomiAI #XiaomiRobotics

Can robot foundation models scale? Xiaomi-Robotics-1 explores this question with over 100,000 hours of real-world manipulation data. Pre-trained on large-scale real-world trajectories and post-trained with cross-embodiment robot data, Xiaomi-Robotics-1 demonstrates consistent scaling across data and model size, strong generalization in unseen environments, and efficient adaptation to new tasks. 🔗 #Robotics #EmbodiedAI #FoundationModels #RobotLearning #XiaomiAI #XiaomiRobotics

Xiaomi Tech

23,920 次观看 • 3 个月前

Brain-computer interfaces (BCI) & Robotics lack large-scale datasets. Fourier Robots is developing BCI-driven robotic rehabilitation training. Imagine a stroke patient thinks of moving a hand, and the robot detects the intent to guide the motion.

Brain-computer interfaces (BCI) & Robotics lack large-scale datasets. Fourier Robots is developing BCI-driven robotic rehabilitation training. Imagine a stroke patient thinks of moving a hand, and the robot detects the intent to guide the motion.

Rohan Paul

91,380 次观看 • 9 个月前

Meet Rove: A Walking Welding #Robot Built for Large-Scale Manufacturing by Space and Technology #Robotics #Innovation #Tech #EmergingTech

Meet Rove: A Walking Welding #Robot Built for Large-Scale Manufacturing by Space and Technology #Robotics #Innovation #Tech #EmergingTech

Ronald van Loon

12,471 次观看 • 3 个月前

Nvidia presents DreamDojo A Generalist Robot World Model from Large-Scale Human Videos

Nvidia presents DreamDojo A Generalist Robot World Model from Large-Scale Human Videos

AK

38,690 次观看 • 8 个月前

German robotics company NEURA Robotics has created NEURA Gym: a large-scale, physical AI gym and training ground where hundreds of robots, including the humanoid 4NE-1, learn through real-world interactions. The company is collaborating with the Neuraverse ecosystem to create the world's largest repository of physical training data, promoting the application of humanoid robots.

German robotics company NEURA Robotics has created NEURA Gym: a large-scale, physical AI gym and training ground where hundreds of robots, including the humanoid 4NE-1, learn through real-world interactions. The company is collaborating with the Neuraverse ecosystem to create the world's largest repository of physical training data, promoting the application of humanoid robots.

CyberRobo

73,731 次观看 • 1 年前

Can robots self-improve by collecting data autonomously🤖? Introducing SOAR: a system for large-scale autonomous data collection 🚀 and autonomous improvement📈of a multi-task language-conditioned policy in diverse scenes without human interventions .

Can robots self-improve by collecting data autonomously🤖? Introducing SOAR: a system for large-scale autonomous data collection 🚀 and autonomous improvement📈of a multi-task language-conditioned policy in diverse scenes without human interventions .

Paul Zhou

47,739 次观看 • 2 年前

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

CyberRobo

18,542 次观看 • 8 个月前

Physical Intelligence's π₀, a general-purpose robot foundation model that combines Internet-scale vision-language pretraining with robot interaction data to execute tasks. They aim "to develop foundation models that can control any robot to perform any task" Autonomous demos:

Physical Intelligence's π₀, a general-purpose robot foundation model that combines Internet-scale vision-language pretraining with robot interaction data to execute tasks. They aim "to develop foundation models that can control any robot to perform any task" Autonomous demos:

The Humanoid Hub

196,941 次观看 • 2 年前

There’s a 400k welder labor shortage in the US alone and the average welder is 55 years old Path Robotics is training robot dogs and other general purpose robots to weld

There’s a 400k welder labor shortage in the US alone and the average welder is 55 years old Path Robotics is training robot dogs and other general purpose robots to weld

Andrew Kang

82,042 次观看 • 3 个月前

Introducing Praxis-1, an open-weight World Action Model that turns our video pre-training expertise into real world control for robots. Robot demonstration data is limited. But video data is infinite. Praxis-1 extends our bet that the best policy models will learn from video, and use that scale to bring embodied intelligence to every industry. Praxis-1 is built on our large-scale video pre-training and real-time infrastructure, and acts as a policy model for robotics developers and researchers that works across any embodiment or environment, no matter how complex. We are actively testing with early partners including Noble Machines, Standard Bots and Ultra, and will be releasing the weights publicly in the coming months. Learn more and request early access at

Introducing Praxis-1, an open-weight World Action Model that turns our video pre-training expertise into real world control for robots. Robot demonstration data is limited. But video data is infinite. Praxis-1 extends our bet that the best policy models will learn from video, and use that scale to bring embodied intelligence to every industry. Praxis-1 is built on our large-scale video pre-training and real-time infrastructure, and acts as a policy model for robotics developers and researchers that works across any embodiment or environment, no matter how complex. We are actively testing with early partners including Noble Machines, Standard Bots and Ultra, and will be releasing the weights publicly in the coming months. Learn more and request early access at

Runway

119,334 次观看 • 1 个月前

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 次观看 • 9 个月前

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Brandon Ong

166,931 次观看 • 2 个月前

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

Physical Intelligence

1,189,291 次观看 • 10 个月前

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵

Robots are the bottleneck in scaling robotics, and learning from human video promises to solve it. But how can chaotic human data ever measure up to sanitized, lab-made teleoperation data? Introducing Do as I Do: establishing a much needed correspondence between human videos and dexterous robot data. Some fun insights below: 🧵

Mahi Shafiullah 🏠🤖

97,211 次观看 • 4 个月前

Tired of collecting demonstrations all day to train your robot? Introducing MimicGen, an autonomous data generation system for robotics. Using just 200 human demos we generated a large multi-task dataset of 50K demos! #CoRL2023 #NVIDIAResearch 👇 🧵 1/

Tired of collecting demonstrations all day to train your robot? Introducing MimicGen, an autonomous data generation system for robotics. Using just 200 human demos we generated a large multi-task dataset of 50K demos! #CoRL2023 #NVIDIAResearch 👇 🧵 1/

Ajay Mandlekar

93,696 次观看 • 3 年前

Train robots. Earn rewards. Base is pioneering the Train-to-Earn model for robotics. With Axis Robotics, anyone can help train robot simulations from a laptop and be rewarded for contributing data. The future of robotics won’t be trained by a handful of companies. It’ll be trained by everyone. PL 漂亮 (✱,✱) showcased this vision at SuperAI, the largest AI event in Asia 👇

Train robots. Earn rewards. Base is pioneering the Train-to-Earn model for robotics. With Axis Robotics, anyone can help train robot simulations from a laptop and be rewarded for contributing data. The future of robotics won’t be trained by a handful of companies. It’ll be trained by everyone. PL 漂亮 (✱,✱) showcased this vision at SuperAI, the largest AI event in Asia 👇

Base APAC

18,980 次观看 • 4 个月前

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

Pascal Bornet

186,881 次观看 • 6 个月前

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

RoboPapers

89,011 次观看 • 1 年前

Robotics is still data starved. Collecting high-quality robot demonstrations remains brutally slow and expensive. Introducing COBALT: A cloud-native teleoperation platform designed for large-scale robot learning. We are democratizing data collection by leveraging the hardware everyone already owns: the smartphone All you need is to download an app (today)! Read on for more!

Robotics is still data starved. Collecting high-quality robot demonstrations remains brutally slow and expensive. Introducing COBALT: A cloud-native teleoperation platform designed for large-scale robot learning. We are democratizing data collection by leveraging the hardware everyone already owns: the smartphone All you need is to download an app (today)! Read on for more!

Animesh Garg

105,038 次观看 • 5 个月前