Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Force is arguably the most overlooked ingredient in modern robot learning. Introducing FACTR 2: it turns any commodity robot into a force-aware system with no force sensors required. Train a tiny force network in <1min with <10mins of data and drop it into any existing teleop pipelines: ✅ Free... show more

Deepak Pathak

31,065 subscribers

41,055 views • 4 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

7 Comments

Nilay Mehrotra4 months ago

F = ma, and yet robot learning has spent years training on the right side of the equation. Demos capture motion the effect while discarding force, the cause For contact-rich tasks that’s learning physics with half the variables hidden. Sensor-free force recovery fixes the right gap

Klajd Lika`4 months ago

Force is the language of Robots. Agreed. For your reference you should check our work for a full analysis and comparison and limitations on estimation via currents, torque sensors and direct sensing at the TCP. Even the “groundtruth force sensing” from Franka or any other estimate from even higher accuracy torque sensing can be hard to deal with. Checkout also our X-Series FT sensors that solves the high cost problems at

Maya N4 months ago

very into this. recovering force, improving demos, then upsampling the parts that actually matter just feels very right.

Prudent AI4 months ago

Intresting 🤔

AgileX Robotics4 months ago

Incredible work! Bringing force awareness to commodity hardware with such minimal data is a massive leap for accessible #EmbodiedAI. We're incredibly proud to see our robotic arms serving as the sandbox for this breakthrough framework. Congrats to the team!

Abhijeet Dhoke1 month ago

Not having force sensor seems like a bad idea. General purpose robot should have force sensor and also tactile sensing of texture of the objects.

Weijie Wang2 months ago

A well-motivated premise and credible approach, but "any robot," "free force sensing," and "strong performance without pretraining" are strong claims whose evidential weight hinges on the paper's force-estimation fidelity and success rates—pending peer verification.

Related Videos

Low-cost teleop systems have democratized robot data collection, but they lack any force feedback, making it challenging to teleoperate contact-rich tasks. Many robot arms provide force information — a critical yet underutilized modality in robot learning. We introduce: 1. 🦾A low-cost, force-feedback-enabled teleop system. 2. 🥊Force-Attending Curriculum Training (FACTR) uses force to improve generalization in complex, contact-rich tasks. 🧵(1/N)

Low-cost teleop systems have democratized robot data collection, but they lack any force feedback, making it challenging to teleoperate contact-rich tasks. Many robot arms provide force information — a critical yet underutilized modality in robot learning. We introduce: 1. 🦾A low-cost, force-feedback-enabled teleop system. 2. 🥊Force-Attending Curriculum Training (FACTR) uses force to improve generalization in complex, contact-rich tasks. 🧵(1/N)

Jason Liu

152,482 views • 1 year ago

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors. We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies. FACTR 2 consists of: 1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors. 2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training. w/ Steven Oh Tony Tao 🧵(1/N)

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors. We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies. FACTR 2 consists of: 1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors. 2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training. w/ Steven Oh Tony Tao 🧵(1/N)

Jason Liu

113,221 views • 4 months ago

1/🧠Humans are the best robot data source — but video alone misses one thing: force. 2/🙁Tactile gloves capture force — but they're costly and block the real touch manipulation depends on. 3/💪Maybe the future of touch lives on your wrist: surface EMG reads the muscles that cause force — tactile sensing without ever touching a tactile sensor. 4/🔥Want a fully open-source framework — hardware + software — to train your own force-aware learn-from-human-data robot policy? 🚀We introduce ForceBand: Learning Forceful Manipulation with sEMG -- bring force into human videos with sEMG, for force-aware manipulation ⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Force Beyond Vision ✦ Free-Hand Force Sensing ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Low-Cost & Easy-to-Implement Let's squeeze every bit of signal out of human data, and let robots feel the force! 🌐 Website: 📄 Paper: 💻 Code: 🎥 Video: 🧵 1/n

1/🧠Humans are the best robot data source — but video alone misses one thing: force. 2/🙁Tactile gloves capture force — but they're costly and block the real touch manipulation depends on. 3/💪Maybe the future of touch lives on your wrist: surface EMG reads the muscles that cause force — tactile sensing without ever touching a tactile sensor. 4/🔥Want a fully open-source framework — hardware + software — to train your own force-aware learn-from-human-data robot policy? 🚀We introduce ForceBand: Learning Forceful Manipulation with sEMG -- bring force into human videos with sEMG, for force-aware manipulation ⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Force Beyond Vision ✦ Free-Hand Force Sensing ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Low-Cost & Easy-to-Implement Let's squeeze every bit of signal out of human data, and let robots feel the force! 🌐 Website: 📄 Paper: 💻 Code: 🎥 Video: 🧵 1/n

Zhi (Leo) Wang

53,320 views • 3 months ago

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 views • 8 months ago

A modular data glove built for robot learning. ENSURING’s Magic Glove uses quick-release sensor modules, combining 15 motion-tracking units, 1,000+ tactile sensing points, and real-time 6-axis force data. The modular design is the interesting part here: sensors can be swapped or reconfigured depending on the task, making it a more flexible interface for teleoperation, dexterous manipulation, and embodied AI data collection.

A modular data glove built for robot learning. ENSURING’s Magic Glove uses quick-release sensor modules, combining 15 motion-tracking units, 1,000+ tactile sensing points, and real-time 6-axis force data. The modular design is the interesting part here: sensors can be swapped or reconfigured depending on the task, making it a more flexible interface for teleoperation, dexterous manipulation, and embodied AI data collection.

Eren Chen

33,006 views • 1 month ago

A lot of robot learning pipelines treat embodiment mismatch as a software problem. Collect data with one device, retarget the motion to another robot, and try to deal with the differences later. TwinDEX flips that idea around. This system is from a Chinese robotics company called X Square. Instead of starting with a generic data-collection device, the team co-designs a wearable three-finger end-effector and a robotic hand as two sides of the same learning system. They align the things that actually matter during manipulation: joint structure, link proportions, contact geometry, surface properties, visual appearance, sensing, and timing. That matters because dexterous manipulation is much more than matching hand poses. A few millimeters of contact error or a different force relationship can completely change how a task is executed. In the collection evaluation, TwinDEX delivered up to 5.3 times the effective throughput of on-robot teleoperation. It’s a useful reminder that scaling robot learning may not only be about bigger models and more data. Sometimes the hardware collecting the data is part of the data problem itself.

A lot of robot learning pipelines treat embodiment mismatch as a software problem. Collect data with one device, retarget the motion to another robot, and try to deal with the differences later. TwinDEX flips that idea around. This system is from a Chinese robotics company called X Square. Instead of starting with a generic data-collection device, the team co-designs a wearable three-finger end-effector and a robotic hand as two sides of the same learning system. They align the things that actually matter during manipulation: joint structure, link proportions, contact geometry, surface properties, visual appearance, sensing, and timing. That matters because dexterous manipulation is much more than matching hand poses. A few millimeters of contact error or a different force relationship can completely change how a task is executed. In the collection evaluation, TwinDEX delivered up to 5.3 times the effective throughput of on-robot teleoperation. It’s a useful reminder that scaling robot learning may not only be about bigger models and more data. Sometimes the hardware collecting the data is part of the data problem itself.

Alamin

48,112 views • 1 month ago

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 views • 1 year ago

Reward AI just came out of stealth and unveiled OM-1, a robot policy learned entirely from humans. No teleoperation, no on-robot data. One Model, Any Body. 1. Capture. A wearable 7-DoF hand (Omnibody Hand) lets people demonstrate manipulation naturally while working, cooking, or sorting. Sensors record tactile, proximity, force and hand pose at full human speed. 2. Learn. OM-1 trains on this human data alone. No teleop, no on-robot experience. Then it runs across bodies from industrial arms to humanoids, and scaling is just adding more human data. 3. Control. A high-frequency control layer, trained with RL in sim, executes those actions on any robot, holding steady through unknown loads like a stuck fridge door. The result: OM-1 picks up a brand-new task from under 30 minutes of data.

The Humanoid Hub

35,672 views • 25 days ago

Force-sensing fingers! 🧤 Stanford researchers just released UMI-FT, a handheld data collection platform that puts compact six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose data. Many manipulation tasks require careful force modulation: too little force and the task fails, too much and you cause damage. But commercial force/torque sensors are expensive, bulky, and fragile, which has limited large-scale force-aware policy learning. UMI-FT changes the economics. The platform uses an iPhone for RGB vision, ultrawide RGB, depth, and pose via ARKit, with each finger sensorized using a CoinFT sensor to capture per-finger wrench information during manipulation. This multimodal data trains an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. The learned policy runs slowest and generates reference targets, while model-based compliance and force controllers provide delicate 6D compliance control and real-time force modulation. They tested on three contact-rich, force-sensitive tasks: whiteboard wiping (locate eraser, grasp, wipe until clean), skewering zucchini (grasp slice firmly, push onto stick until punctured), and lightbulb insertion (grasp bulb, align bayonet pin with socket slit, insert while overcoming spring force, rotate to light up). The results are clear. Policies without compliance struggle to modulate contact force and trigger safety faults from excessive force. Policies without force sensing fail to grasp unseen objects or resist reaction forces, causing slippage. Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Force-sensing fingers! 🧤 Stanford researchers just released UMI-FT, a handheld data collection platform that puts compact six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose data. Many manipulation tasks require careful force modulation: too little force and the task fails, too much and you cause damage. But commercial force/torque sensors are expensive, bulky, and fragile, which has limited large-scale force-aware policy learning. UMI-FT changes the economics. The platform uses an iPhone for RGB vision, ultrawide RGB, depth, and pose via ARKit, with each finger sensorized using a CoinFT sensor to capture per-finger wrench information during manipulation. This multimodal data trains an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. The learned policy runs slowest and generates reference targets, while model-based compliance and force controllers provide delicate 6D compliance control and real-time force modulation. They tested on three contact-rich, force-sensitive tasks: whiteboard wiping (locate eraser, grasp, wipe until clean), skewering zucchini (grasp slice firmly, push onto stick until punctured), and lightbulb insertion (grasp bulb, align bayonet pin with socket slit, insert while overcoming spring force, rotate to light up). The results are clear. Policies without compliance struggle to modulate contact force and trigger safety faults from excessive force. Policies without force sensing fail to grasp unseen objects or resist reaction forces, causing slippage. Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

12,868 views • 8 months ago

What if a dexterous robot could learn a 20+ step chemistry experiment without a single on-robot training demo? Meet TwinDEX: a pair of co-designed, three-finger, nine-DoF dexterous manipulation interface: one wearable for data collection, one for robot deployment. The twinned design shares identical kinematics, contact surfaces, visual appearance, and sensors across collection and deployment — keeping observations and actions aligned end to end. Trained from scratch on only a few hundred wearable demonstrations - with zero on-robot training or intervention data - TwinDEX completed a standardized chemistry experiment involving tool switches, fine force control, and bimanual coordination. Robot-free data showed comparable learning efficiency on the multi-task evaluation, TwinDEX delivered 5.3 times effective throughput than on-robot teleoperation. TwinDEX demonstrates that high-quality robot-free data can fully substitute for on-robot teleoperation data on challenging dexterous tasks — removing the dependency on real-robot hardware that has been the central bottleneck to scaling dexterous manipulation data. This was the proof-it phase. Now comes scale: what emerges at tens of thousands, or millions, of episodes? Watch the demo and read the technical blog: #TwinDEX #Robotics #EmbodiedAI #DexterousManipulation

What if a dexterous robot could learn a 20+ step chemistry experiment without a single on-robot training demo? Meet TwinDEX: a pair of co-designed, three-finger, nine-DoF dexterous manipulation interface: one wearable for data collection, one for robot deployment. The twinned design shares identical kinematics, contact surfaces, visual appearance, and sensors across collection and deployment — keeping observations and actions aligned end to end. Trained from scratch on only a few hundred wearable demonstrations - with zero on-robot training or intervention data - TwinDEX completed a standardized chemistry experiment involving tool switches, fine force control, and bimanual coordination. Robot-free data showed comparable learning efficiency on the multi-task evaluation, TwinDEX delivered 5.3 times effective throughput than on-robot teleoperation. TwinDEX demonstrates that high-quality robot-free data can fully substitute for on-robot teleoperation data on challenging dexterous tasks — removing the dependency on real-robot hardware that has been the central bottleneck to scaling dexterous manipulation data. This was the proof-it phase. Now comes scale: what emerges at tens of thousands, or millions, of episodes? Watch the demo and read the technical blog: #TwinDEX #Robotics #EmbodiedAI #DexterousManipulation

X Square Robot

1,346,470 views • 1 month ago

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes even*cheaper* with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes evencheaper with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

Abhishek Gupta

13,358 views • 1 year ago

How precise can a robot get… without ever seeing the object move? Turns out, just a geometric model is enough. A robot hit 73 precise 3D goals in a row, using only a geometric model. No demonstrations. No imitation learning. No fancy vision models. The secret? Sample where the robot could make contact. Let local contact-implicit control take over from there. That’s it. ✅ Real-time performance ✅ Natural contact making and breaking ✅ Works on tricky tasks like pushing and rotating This approach mixes global planning with local MPC. Fast. Accurate. Sample-efficient. Thanks for sharing, Michael Posa ! 📍Watch it in action: Paper:

How precise can a robot get… without ever seeing the object move? Turns out, just a geometric model is enough. A robot hit 73 precise 3D goals in a row, using only a geometric model. No demonstrations. No imitation learning. No fancy vision models. The secret? Sample where the robot could make contact. Let local contact-implicit control take over from there. That’s it. ✅ Real-time performance ✅ Natural contact making and breaking ✅ Works on tricky tasks like pushing and rotating This approach mixes global planning with local MPC. Fast. Accurate. Sample-efficient. Thanks for sharing, Michael Posa ! 📍Watch it in action: Paper:

Ilir Aliu

19,012 views • 1 year ago

Most AI agents force you into a new browser. Meet Composite, it isn't just a hosted browser session in the cloud, or a completely separate browser. Composite actually does the clicking, typing, and navigation inside your real tabs, in your existing browser. - Uses your existing browser, so there’s no migration, you keep your bookmarks, extensions, workflows - No lock-in. It works with any site, any AI model, you’re in control. - Composite does the clicks, types, and navigation for you. In your tabs. No migration. No lock-in. No learning curve. Download and start now:

Most AI agents force you into a new browser. Meet Composite, it isn't just a hosted browser session in the cloud, or a completely separate browser. Composite actually does the clicking, typing, and navigation inside your real tabs, in your existing browser. - Uses your existing browser, so there’s no migration, you keep your bookmarks, extensions, workflows - No lock-in. It works with any site, any AI model, you’re in control. - Composite does the clicks, types, and navigation for you. In your tabs. No migration. No lock-in. No learning curve. Download and start now:

The AI Colony

48,585 views • 1 year ago

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,867 views • 4 months ago

First demo of reproducing force feedback from the FACTR 2 paper It relies on torque and current sensing which the feetechs don’t handle as well due to large gear ratios so it’s not perfect. Not sure if limits are in hw or we can do better with core changes & more data

First demo of reproducing force feedback from the FACTR 2 paper It relies on torque and current sensing which the feetechs don’t handle as well due to large gear ratios so it’s not perfect. Not sure if limits are in hw or we can do better with core changes & more data

Ville🤖

19,216 views • 2 months ago

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

18,813 views • 8 months ago

Quadrupeds are fast. Agile. Great at locomotion. But can they manipulate? A new approach from Carnegie Mellon University, Google DeepMind, and Bosch is teaching quadrupedal robots to do more than walk, they’re learning to interact. It’s called Human2LocoMan: a system that uses human data to pretrain robot policies before finetuning on real hardware. The result? A four-legged robot that can walk, carry, organize, scoop, and sort with both single and dual-arm control. By pretraining on human motion, they cut the amount of robot data in half—while improving success rates by over 80% in unfamiliar environments. Their Modularized Cross-Embodiment Transformer (MXT) learns from both human and robot demonstrations, then generalizes those skills to physical tasks—no hardcoded behaviors required. It’s locomotion and manipulation. A quadruped that can walk and clean up after itself?

Quadrupeds are fast. Agile. Great at locomotion. But can they manipulate? A new approach from Carnegie Mellon University, Google DeepMind, and Bosch is teaching quadrupedal robots to do more than walk, they’re learning to interact. It’s called Human2LocoMan: a system that uses human data to pretrain robot policies before finetuning on real hardware. The result? A four-legged robot that can walk, carry, organize, scoop, and sort with both single and dual-arm control. By pretraining on human motion, they cut the amount of robot data in half—while improving success rates by over 80% in unfamiliar environments. Their Modularized Cross-Embodiment Transformer (MXT) learns from both human and robot demonstrations, then generalizes those skills to physical tasks—no hardcoded behaviors required. It’s locomotion and manipulation. A quadruped that can walk and clean up after itself?

Lukas Ziegler

61,610 views • 1 year ago