Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

🤔 Can we train one policy to control a wide range of robots, from drones to quadrupeds, navigators to bimanual manipulators, and more? 🦾Introducing CrossFormer: a single policy that can perform manipulation, navigation, aviation, and locomotion:

riadoshi21's profile picture

Ria Doshi

1,029 subscribers

85,220 görüntüleme • 2 yıl önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

13 Yorum

Ria Doshi profil fotoğrafı
Ria Doshi2 yıl önce

Cross-embodiment policy learning poses many challenges. Robots may have different control control frequencies, observation spaces (proprioception vs images), and action spaces (2D navigation vs 14D bimanual manipulation)

Ria Doshi profil fotoğrafı
Ria Doshi2 yıl önce

The core of our method is a transformer-based policy. By framing cross-embodiment training as a sequence-to-sequence problem, we’re able to flexibly adapt our model to robots of varying observation and action spaces

Ria Doshi profil fotoğrafı
Ria Doshi2 yıl önce

Our model’s flexible nature allows us to outperform and at least match state-of-the-art performance across six embodiments, ranging from bimanual manipulators, navigators, quadrupeds, and more!

Ria Doshi profil fotoğrafı
Ria Doshi2 yıl önce

We also outperform the best prior cross-embodiment learning work by 3x. Prior work aligns observation & action spaces for navigation and manipulation; we hypothesize that this alignment can sometimes hinder performance on more complex tasks

Ria Doshi profil fotoğrafı
Ria Doshi2 yıl önce

Check out our website and code for more details on CrossFormer! Website: Paper: Code:

Ria Doshi profil fotoğrafı
Ria Doshi2 yıl önce

Really grateful and lucky to have worked with some incredible people on this project: co-lead @HomerWalke and collaborators @oier_mees @SudeepDasari @svlevine

Brett Adcock profil fotoğrafı
Brett Adcock2 yıl önce

Cool

BensenHsu profil fotoğrafı
BensenHsu2 yıl önce

The paper aims to train a single policy that can control a diverse set of robot embodiments, including single and bimanual robot arms, ground navigation robots, quadcopters, and quadrupeds. This is a challenging task as robots can have widely varying sensors, actuators, and control frequencies. The researchers find that their CrossFormer policy matches the performance of the same architecture trained only on the target robot's data, as well as the best prior method in each evaluation setting. CrossFormer outperforms the prior state-of-the-art in cross-embodiment learning by a large margin, without requiring manual alignment of observation and action spaces. full paper:

Haitham Bou Ammar profil fotoğrafı
Haitham Bou Ammar2 yıl önce

@davide_tateo

Zheyuan Hu profil fotoğrafı
Zheyuan Hu2 yıl önce

Super cool work, congrats Ria!

Quadruped profil fotoğrafı
Quadruped2 yıl önce

Very cool.

Krish Dasgupta profil fotoğrafı
Krish Dasgupta2 yıl önce

Absolutely brilliant. Will check this out

Haoran He profil fotoğrafı
Haoran He2 yıl önce

Interesting work👍

Benzer Videolar

🤔 Can we train one VLA policy to control multi-robot teams without any explicit communication? ✨ Introducing CHORUS: a single policy for decentralized, multi-embodiment collaboration 🧵⬇️
0:32

🤔 Can we train one VLA policy to control multi-robot teams without any explicit communication? ✨ Introducing CHORUS: a single policy for decentralized, multi-embodiment collaboration 🧵⬇️

Ria Doshi

61,245 görüntüleme • 3 ay önce

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at
0:31

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at

RAI Institute

13,560 görüntüleme • 1 yıl önce

Cross-embodied robot policies hold the promise of one policy to control all robots. But how far does transfer go? In new work, we study positive transfer between *manipulation* & *navigation* and show that nav data helps manipulation, and vice versa! 🧵 👇
0:53

Cross-embodied robot policies hold the promise of one policy to control all robots. But how far does transfer go? In new work, we study positive transfer between *manipulation* & *navigation* and show that nav data helps manipulation, and vice versa! 🧵 👇

Sergey Levine

42,721 görüntüleme • 2 yıl önce

Truly general-purpose robots must be able to navigate spaces they have never seen before. ⦿ Skild Brain enables end-to-end autonomous locomotion from raw vision and joint inputs, without mapping or pre-planning. ⦿ The model adapts in real time to new terrain such as stairs, gaps, and clutter, reacting instinctively like a human. There is no “stair mode” or “step-over mode”: the transition between walking behaviors is continuous and instinctive. ⦿ A single neural policy supports a wide range of locomotion tasks and can transfer across different robot platforms.
4:37

Truly general-purpose robots must be able to navigate spaces they have never seen before. ⦿ Skild Brain enables end-to-end autonomous locomotion from raw vision and joint inputs, without mapping or pre-planning. ⦿ The model adapts in real time to new terrain such as stairs, gaps, and clutter, reacting instinctively like a human. There is no “stair mode” or “step-over mode”: the transition between walking behaviors is continuous and instinctive. ⦿ A single neural policy supports a wide range of locomotion tasks and can transfer across different robot platforms.

The Humanoid Hub

28,439 görüntüleme • 1 yıl önce

Introducing 2️⃣ new AI systems for robotics: 🤖 ALOHA Unleashed to perform two-armed manipulation tasks 🦾 DemoStart to control a multi-fingered robotic hand They learned to tackle a range of actions requiring dexterity. Here's how. 🧵
0:09

Introducing 2️⃣ new AI systems for robotics: 🤖 ALOHA Unleashed to perform two-armed manipulation tasks 🦾 DemoStart to control a multi-fingered robotic hand They learned to tackle a range of actions requiring dexterity. Here's how. 🧵

Google DeepMind

297,345 görüntüleme • 2 yıl önce

Robots need to be able to apply pressure and make contact with objects as needed in order to accomplish their tasks. From compliance to working safely around humans to whole-body manipulation of heavy objects, combining force and position control can dramatically expand the capabilities of robots. This is especially true for legged robots, which have so much ability to exert forces on the world around them. But how do we train robots which can do this? Baoxiong Jia tells us more in our discussion of his team’s recent, Best Paper Award winning work on learning a unified policy for position and force control, called UniFP. To learn more, watch Episode #49 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton.
50:15

Robots need to be able to apply pressure and make contact with objects as needed in order to accomplish their tasks. From compliance to working safely around humans to whole-body manipulation of heavy objects, combining force and position control can dramatically expand the capabilities of robots. This is especially true for legged robots, which have so much ability to exert forces on the world around them. But how do we train robots which can do this? Baoxiong Jia tells us more in our discussion of his team’s recent, Best Paper Award winning work on learning a unified policy for position and force control, called UniFP. To learn more, watch Episode #49 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton.

RoboPapers

44,825 görüntüleme • 9 ay önce

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:
3:05

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Zhengyi “Zen” Luo

92,214 görüntüleme • 2 yıl önce

Ground drones in service with the 77th Airborne Brigade Drones perform a wide range of tasks: from logistics and minelaying to evacuation of the wounded and fire cover.
1:14

Ground drones in service with the 77th Airborne Brigade Drones perform a wide range of tasks: from logistics and minelaying to evacuation of the wounded and fire cover.

Special Kherson Cat 🐈🇺🇦

19,789 görüntüleme • 7 ay önce

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from Stanford University designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)
1:16

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from Stanford University designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)

Cheng Chi

439,927 görüntüleme • 2 yıl önce

Quadrupeds are fast. Agile. Great at locomotion. But can they manipulate? A new approach from Carnegie Mellon University, Google DeepMind, and Bosch is teaching quadrupedal robots to do more than walk, they’re learning to interact. It’s called Human2LocoMan: a system that uses human data to pretrain robot policies before finetuning on real hardware. The result? A four-legged robot that can walk, carry, organize, scoop, and sort with both single and dual-arm control. By pretraining on human motion, they cut the amount of robot data in half—while improving success rates by over 80% in unfamiliar environments. Their Modularized Cross-Embodiment Transformer (MXT) learns from both human and robot demonstrations, then generalizes those skills to physical tasks—no hardcoded behaviors required. It’s locomotion and manipulation. A quadruped that can walk and clean up after itself?
1:21

Quadrupeds are fast. Agile. Great at locomotion. But can they manipulate? A new approach from Carnegie Mellon University, Google DeepMind, and Bosch is teaching quadrupedal robots to do more than walk, they’re learning to interact. It’s called Human2LocoMan: a system that uses human data to pretrain robot policies before finetuning on real hardware. The result? A four-legged robot that can walk, carry, organize, scoop, and sort with both single and dual-arm control. By pretraining on human motion, they cut the amount of robot data in half—while improving success rates by over 80% in unfamiliar environments. Their Modularized Cross-Embodiment Transformer (MXT) learns from both human and robot demonstrations, then generalizes those skills to physical tasks—no hardcoded behaviors required. It’s locomotion and manipulation. A quadruped that can walk and clean up after itself?

Lukas Ziegler

61,610 görüntüleme • 1 yıl önce

We open-sourced all components of the HTD whole-body controller (WBC): -Play our WBC on your browser: -Main HTD codebase entry: (data collection and HTD policy soon!) Highlighted features: -Lightweight decoupled WBC that can be trained on a single GPU -A stable control base for humanoid loco-manipulation (in a mobile-manipulation style) -Support a wide range of body poses including extreme ones -Isaac Sim/Lab training, zero-shot deployment
1:17

We open-sourced all components of the HTD whole-body controller (WBC): -Play our WBC on your browser: -Main HTD codebase entry: (data collection and HTD policy soon!) Highlighted features: -Lightweight decoupled WBC that can be trained on a single GPU -A stable control base for humanoid loco-manipulation (in a mobile-manipulation style) -Support a wide range of body poses including extreme ones -Isaac Sim/Lab training, zero-shot deployment

Yaru Niu

22,474 görüntüleme • 1 ay önce

Mark Carney: "We can't control the trade policy of the United States. We recognize that policy has fundamentally changed from the policy in the 1980s, 1990s, 2000s, and it's a situation where the US has tariffs against every one of their trading partners ... what we can control is developing new partnerships and opportunities including with the economic giants of Asia."
1:36

Mark Carney: "We can't control the trade policy of the United States. We recognize that policy has fundamentally changed from the policy in the 1980s, 1990s, 2000s, and it's a situation where the US has tariffs against every one of their trading partners ... what we can control is developing new partnerships and opportunities including with the economic giants of Asia."

Aaron Rupar

2,035,745 görüntüleme • 11 ay önce

The most frustrating part of imitation learning is collecting huge amounts of teleop data. But why teleop robots when robots can learn by watching us? Introducing Point Policy, a novel framework that enables robots to learn from human videos without any teleop, sim2real, or RL.
0:40

The most frustrating part of imitation learning is collecting huge amounts of teleop data. But why teleop robots when robots can learn by watching us? Introducing Point Policy, a novel framework that enables robots to learn from human videos without any teleop, sim2real, or RL.

Siddhant Haldar

69,056 görüntüleme • 1 yıl önce

🦾 Our robot hand can rotate objects over 6+ axes in the real-world! Introducing RotateIt (CoRL 2023), a Sim-to-Real policy that can rotate many objects over many axes, using vision and touch! Check it out: Paper: #CoRL2023
0:25

🦾 Our robot hand can rotate objects over 6+ axes in the real-world! Introducing RotateIt (CoRL 2023), a Sim-to-Real policy that can rotate many objects over many axes, using vision and touch! Check it out: Paper: #CoRL2023

Haozhi Qi

51,041 görüntüleme • 3 yıl önce

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵
0:30

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

Chelsea Finn

264,494 görüntüleme • 3 yıl önce

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.
0:30

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.

Space and Technology

31,172 görüntüleme • 12 gün önce

📢📢📢 Excited to release ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR25). 🤏🤙✌️With ManipTrans, we can transfer dexterous manipulation skills into robotic hands in simulation and deploy them on a real robot, using a residual policy learned for dex manipulation. 🤖🤖🤖The video below illustrates how the MoCap data can be transferred to Inspire, Shadow, Xhand, Allegro, and Mano. With ManipTrans, we can scale up dex manip data greatly with minimal effort. For more details, please check our -webpage: -code: -huggingface:
1:59

📢📢📢 Excited to release ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning (CVPR25). 🤏🤙✌️With ManipTrans, we can transfer dexterous manipulation skills into robotic hands in simulation and deploy them on a real robot, using a residual policy learned for dex manipulation. 🤖🤖🤖The video below illustrates how the MoCap data can be transferred to Inspire, Shadow, Xhand, Allegro, and Mano. With ManipTrans, we can scale up dex manip data greatly with minimal effort. For more details, please check our -webpage: -code: -huggingface:

Siyuan Huang

20,983 görüntüleme • 1 yıl önce

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D
0:20

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,892 görüntüleme • 1 yıl önce