Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🤖 Robotics often faces a chicken and egg problem: no web-scale robot data for training (unlike CV or NLP) b/c robots aren't deployed yet & vice-versa. Introducing VRB: Use large-scale human videos to train a general-purpose affordance model to jumpstart any robotics paradigm!

Deepak Pathak

30,607 subscribers

70,712 views • 3 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

10 Comments

Deepak Pathak3 years ago

🌐 Given a new scene, our general-purpose VRB model predicts all the locations *where* a robot can manipulate objects and *how* should it move post-manipulation. 💡Qs: 1) What are affordances? 2) How to get large-scale data? 3) How does this help robots?

Deepak Pathak3 years ago

🧩 In computer vision, there is a long line of work in affordances (Gibson 1966, 1979). However, what's the best way to define them for robotics? Our solution: interaction points & post-contact trajectories, allowing for seamless integration with almost any robot learning setup.

Deepak Pathak3 years ago

🔧🔧 How do we extract affordances from humans? We estimate hand poses & hand-object interaction points to extract contact pts + interaction direction (thanks to @DandanShan_, David Fouhey's 100DoH), and map them back to frames where humans arent present to avoid embodiment gap.

Deepak Pathak3 years ago

How to use this model for different robot learning paradigms? We show 4 different paradigms. 1. Offline Data Collection 📊 Instead of using teleoperation or scripted policies, VRB can allow for an automatic collection of good quality interaction-rich data for policy learning.

Deepak Pathak3 years ago

2. Bootstrapping Exploration 🔍 Robots can utilize the predicted affordances from VRB to explore their environment more intelligently, leading to the discovery of novel and effective ways to manipulate objects.

Deepak Pathak3 years ago

3. Goal-Conditioned Learning 🎯 Can we use our affordance model to iteratively improve performance? We train goal-oriented policies using actions sampled from VRB. We prune this action distribution using the distance to the goal during the training process (using WHIRL RSS'22).

Deepak Pathak3 years ago

4. Action Space Reparameterization 🔄 We can also treat our affordances as an action space itself. It helps accelerate online RL eliminating the need for task-specific primitives. This enables DQN to learn manipulation tasks in under 30 mins!

Deepak Pathak3 years ago

Using internet-scale videos is a promising approach to tackling the data problem in robotics and we hope VRB is a strong step toward that! Led by @shikharbahl, @mendonca_rl, @lchen915, @unnatjain2010 CVPR 2023 Paper: Website: 9/9

Murtaza Dalal3 years ago

Awesome work by my colleagues @shikharbahl, @mendonca_rl, @lchen915! Extracting affordances from human video enables robots to efficiently solve a wide array of manipulation tasks in the real world 🤖. Excited to see where this goes next!

Siyuan HUANG3 years ago

Great ideas and a very interesting presentation. :-) I have checked your website but find the dataset is coming soon, would that be available soon?

Related Videos

Brain-computer interfaces (BCI) & Robotics lack large-scale datasets. Fourier Robots is developing BCI-driven robotic rehabilitation training. Imagine a stroke patient thinks of moving a hand, and the robot detects the intent to guide the motion.

Brain-computer interfaces (BCI) & Robotics lack large-scale datasets. Fourier Robots is developing BCI-driven robotic rehabilitation training. Imagine a stroke patient thinks of moving a hand, and the robot detects the intent to guide the motion.

Rohan Paul

91,321 views • 7 months ago

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

Humanoid robotics is entering a new phase： Xiaomi Robotics 0 (XR-0) explores learning manipulation directly from large-scale human videos — aligning video understanding with robot embodiment to generate executable actions. Watch humans->learn representations ->map to robot control. Robot data is scarce. Internet video is not. If video pretraining transfers reliably into embodied control, the scaling law of robotics changes. Deployment speed becomes a function of model alignment, not just hardware iteration. We’re moving from engineered skills to learned behavior. That’s a structural shift. project:

CyberRobo

18,356 views • 6 months ago

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 views • 7 months ago

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Today we're launching Hebbian Robotics (YC S26) Hebbian Robotics (YC S26) . We build APIs for evaluating data quality, without training a robotics model. Data vendors use our API to search and analyze Physical AI data at scale, getting on-demand quality signals and metrics without managing infrastructure. High-quality data is the bottleneck to the next frontier of intelligence in robotics. Buyers want data they can trust. Data providers are focused on operational excellence. They don't have the bandwidth to understand the quality of their data at scale. We're fixing this. Vendors use our APIs to understand the impact of their decisions on their data quality: - Buyers demanding per-collector hour limits? Prove compliance while we stay ahead of the workarounds. - Changed your collection SOPs? Monitor for quality drift and regression. - Suspect duplicate samples? Verify with a single API call. Before Hebbian, I trained AI models for industrial cleaning robots, and Kingston Kuan spent years building large-scale infrastructure at Jane Street.

Brandon Ong

166,522 views • 1 month ago

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

Physical Intelligence

1,186,779 views • 9 months ago

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

Pascal Bornet

186,745 views • 5 months ago

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

Robots has a data problem, in that robotics data is rare. While human video is quite common, it’s not usually directly usable for robots for a variety of reasons, most significantly that it’s missing explicit, accurate robot actions. Instead, Jeremy Collins proposes that we predict keypoint trajectories — basically, how any given point in an object will move as a robot performs a task. This lets us use action-free human video to train robot skills. Learn more by watching Episode #37 of RoboPapers with Michael Cho - Rbt/Acc and Chris Paxton now.

RoboPapers

89,011 views • 10 months ago

Robotics is still data starved. Collecting high-quality robot demonstrations remains brutally slow and expensive. Introducing COBALT: A cloud-native teleoperation platform designed for large-scale robot learning. We are democratizing data collection by leveraging the hardware everyone already owns: the smartphone All you need is to download an app (today)! Read on for more!

Robotics is still data starved. Collecting high-quality robot demonstrations remains brutally slow and expensive. Introducing COBALT: A cloud-native teleoperation platform designed for large-scale robot learning. We are democratizing data collection by leveraging the hardware everyone already owns: the smartphone All you need is to download an app (today)! Read on for more!

Animesh Garg

104,496 views • 3 months ago

Robots can already fold laundry, make espresso, clean kitchens, and assemble things. The harder problem is getting them to do those tasks reliably, for long periods of time, without a human babysitting them. At Startup School 2026, Physical Intelligence cofounder Chelsea Finn explains what it takes to build general-purpose robots that work in the real world. She shares how reinforcement learning pushed robot throughput up 2x, how their systems can run autonomously for hours, and why she believes robotics is entering its GPT era: moving from specialized models toward general-purpose systems that can work across tasks, robots, and environments. 00:00 — The State of Physical Intelligence 01:23 — What It Takes to Make Robots Useful 05:11 — The Reliability Problem 07:43 — Reinforcement Learning for Robotics 09:35 — Learning From Failures 12:43 — Training Robots to Improve Themselves 14:21 — Can a Robot Work for 13 Hours Straight? 17:36 — Why Robots Need Memory 21:22 — Building a General-Purpose Robot 25:02 — From Fine-Tuning to Out-of-the-Box Models 27:35 — Training on All the Data 30:20 — One Model That Beats the Specialists 31:21 — Compositional Generalization 37:49 — The GPT Era of Robotics 39:49 — Q&A

Robots can already fold laundry, make espresso, clean kitchens, and assemble things. The harder problem is getting them to do those tasks reliably, for long periods of time, without a human babysitting them. At Startup School 2026, Physical Intelligence cofounder Chelsea Finn explains what it takes to build general-purpose robots that work in the real world. She shares how reinforcement learning pushed robot throughput up 2x, how their systems can run autonomously for hours, and why she believes robotics is entering its GPT era: moving from specialized models toward general-purpose systems that can work across tasks, robots, and environments. 00:00 — The State of Physical Intelligence 01:23 — What It Takes to Make Robots Useful 05:11 — The Reliability Problem 07:43 — Reinforcement Learning for Robotics 09:35 — Learning From Failures 12:43 — Training Robots to Improve Themselves 14:21 — Can a Robot Work for 13 Hours Straight? 17:36 — Why Robots Need Memory 21:22 — Building a General-Purpose Robot 25:02 — From Fine-Tuning to Out-of-the-Box Models 27:35 — Training on All the Data 30:20 — One Model That Beats the Specialists 31:21 — Compositional Generalization 37:49 — The GPT Era of Robotics 39:49 — Q&A

Y Combinator

111,276 views • 1 month ago

Yann LeCun says that within a year to 18 months, we'll have a general method for training hierarchical world models These models would learn from video and real-world data, then help plan actions in robotics, healthcare, and other areas "then scale them toward a universal world model"

Yann LeCun says that within a year to 18 months, we'll have a general method for training hierarchical world models These models would learn from video and real-world data, then help plan actions in robotics, healthcare, and other areas "then scale them toward a universal world model"

Haider.

201,384 views • 3 months ago

Reasoning is central to purposeful action. Today we introduce MolmoAct — a fully open Action Reasoning Model (ARM) for robotics. Grounded in large-scale pre-training with action reasoning data, every predicted action is interpretable and user-steerable via visual trace. We are open-sourcing everything!

Reasoning is central to purposeful action. Today we introduce MolmoAct — a fully open Action Reasoning Model (ARM) for robotics. Grounded in large-scale pre-training with action reasoning data, every predicted action is interpretable and user-steerable via visual trace. We are open-sourcing everything!

Jiafei Duan

99,944 views • 1 year ago

1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSource

1/5 🚀 Thrilled to open-source OSCAR 🤖 — an action-conditioned world model for robotics, led by the visiting student in my group Zhuoyuan Wu! It generalizes across different robot embodiments with precise action controllability. All trained on a single GH200 GPU, and outperforms existing open-sourced baselines, which have larger model capacity and need more compute. Everything is public, including training data. 📄 Paper: 🌐 Project: 💻 Code: 🤗 Robot data: 🤗 Human data: 🤗 Weights: #Robotics #WorldModels #AI #OpenSource

Jun Gao

104,967 views • 3 months ago

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

MTS

17,637 views • 1 month ago

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 views • 6 months ago

Robotics is entering the mainstream! 👀 Let's accept this fact. Consumptionism – stonks. Possibilities (HW + SW) – stonks. Demographics - not stonks. Modern robots are often designed to interact with our legacy world, whether it's a household, a factory, or a shipyard. That means they often need to work with existing machines, tools, and spaces. In short: retrofitting. Improving old machines with new technologies without major interference in their operation. Modern way of integrating things. You don't have to change the entire environment to deploy a robot. The robot will adapt, just as human would when freshly hired. The promise is to have an army of robots to help with the world designed for us. I also believe that general purpose robots, are not a direct competition for single purpose machines (like a highly efficient specialized packaging machine, for example). It's like adding a robot to this carriage. Specialized machine could be a car ;) ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robotics is entering the mainstream! 👀 Let's accept this fact. Consumptionism – stonks. Possibilities (HW + SW) – stonks. Demographics - not stonks. Modern robots are often designed to interact with our legacy world, whether it's a household, a factory, or a shipyard. That means they often need to work with existing machines, tools, and spaces. In short: retrofitting. Improving old machines with new technologies without major interference in their operation. Modern way of integrating things. You don't have to change the entire environment to deploy a robot. The robot will adapt, just as human would when freshly hired. The promise is to have an army of robots to help with the world designed for us. I also believe that general purpose robots, are not a direct competition for single purpose machines (like a highly efficient specialized packaging machine, for example). It's like adding a robot to this carriage. Specialized machine could be a car ;) ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

14,396 views • 7 months ago

JUST IN: Dyna Robotics just published one of the most important research papers in robotics this year. It could fundamentally change how robot foundation models are trained. A scaling law that transfers from human video to robot performance. Dyna-2 is out and it's 🔥 Here's what that means in plain terms. Dyna-2 was pre-trained on ONE MILLION hours of egocentric human video, 170 years of continuous human experience, cooking, folding, assembling, cleaning. And as that human data scaled, robot performance improved. Predictably. Monotonically. Across 39 tasks on two different robot embodiments the model had never seen. → 1,000 hours pre-training → 20% normalised task performance → 10,000 hours → 28% → 100,000 hours → 45% → 1,000,000 hours → 53% Human video exists at effectively unlimited scale. Every cook, every factory worker, every craftsperson wearing a camera is generating training data for future robots. But the finding that stunned even the researchers, world modeling is what makes the transfer work. A model trained to predict future video AND actions massively outperforms one trained on actions alone. Video is the new scaling axis for robotics. One more jaw-dropping data point. 13 minutes of teleoperation data was enough to fine-tune Dyna-2 to open a bottle cap using two five-fingered robot hands. The robots are coming, and they're learning from us directly :D Read more here: Congrats Jason Ma and team! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

JUST IN: Dyna Robotics just published one of the most important research papers in robotics this year. It could fundamentally change how robot foundation models are trained. A scaling law that transfers from human video to robot performance. Dyna-2 is out and it's 🔥 Here's what that means in plain terms. Dyna-2 was pre-trained on ONE MILLION hours of egocentric human video, 170 years of continuous human experience, cooking, folding, assembling, cleaning. And as that human data scaled, robot performance improved. Predictably. Monotonically. Across 39 tasks on two different robot embodiments the model had never seen. → 1,000 hours pre-training → 20% normalised task performance → 10,000 hours → 28% → 100,000 hours → 45% → 1,000,000 hours → 53% Human video exists at effectively unlimited scale. Every cook, every factory worker, every craftsperson wearing a camera is generating training data for future robots. But the finding that stunned even the researchers, world modeling is what makes the transfer work. A model trained to predict future video AND actions massively outperforms one trained on actions alone. Video is the new scaling axis for robotics. One more jaw-dropping data point. 13 minutes of teleoperation data was enough to fine-tune Dyna-2 to open a bottle cap using two five-fingered robot hands. The robots are coming, and they're learning from us directly :D Read more here: Congrats Jason Ma and team! ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

23,576 views • 1 month ago

Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence Tutor is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz, and Joe McCalmon, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence Tutor is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz, and Joe McCalmon, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

36,041 views • 3 months ago

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

59,835 views • 8 months ago