Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

🕸️ Introducing SPIDER — Scalable Physics-Informed Dexterous Retargeting! A dynamically feasible, cross-embodiment retargeting framework for BOTH humanoids 🤖 and dexterous hands ✋. From human motion → sim → real robots, at scale. 🔗 Website: 🧵 1/n

Chaoyi Pan

1,105 subscribers

66,077 Aufrufe • vor 11 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

21 Kommentare

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

🧵 2/n Retargeting human motion is hard: embodiment gap, missing forces, noisy data. IK fails ❌, RL is costly ❌. SPIDER converts kinematic human demos into dynamically feasible robot trajectories using physics-informed sampling + virtual contact guidance. 👇 Pipeline

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

🧵 3/n Because SPIDER is grounded with physics, the generated motions are directly executable on real hardware—no tuning. Zero robot demos. No RL training. Just human → physics → real robot.

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

🧵 4/n SPIDER supports diverse embodiments: 🤖 humanoid + ✋ dexterous hand 📦 2.4M robot-feasible frames 🤖 9 embodiments 🧊 103 objects ⏱️ 800 hrs of data

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

🧵 5/n SPIDER is also a data engine: We vary object size, states, and scenes to produce massive, physics-valid training data.

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

🧵 6/n For more details, please refer to: 🔗 Website: 📄 Paper: Huge thanks to the team at FAIR + CMU! We hope SPIDER pushes scalable, physics-grounded retargeting forward 🕷️🤖

Profilbild von C. Zhang

C. Zhangvor 11 Monaten

jc-bao🦾

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

@ChongZitaZhang Thanks Chong!

Profilbild von Jingyun Yang

Jingyun Yangvor 11 Monaten

Amazing work. Congrats @ChaoyiPan!

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

Thanks, Jingyun!

Profilbild von Reza Sayar

Reza Sayarvor 11 Monaten

@pablovelagomez1 👀

Profilbild von Weishuai Zeng

Weishuai Zengvor 11 Monaten

Really impressive! When kinematic-based retargeter fails to preserve necessary physical information(like contact during interaction) and neural-based retargeter is hard to directly train, such physics-aware retarger becomes as the best option. Looking forward to the release!

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

Thanks Weishuai!

Profilbild von Wenli Xiao

Wenli Xiaovor 11 Monaten

Very impressive work, congrats jc-bao!

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

Thank you Wenli!

Profilbild von Shucheng Kang

Shucheng Kangvor 11 Monaten

Congratulations!

Profilbild von Guanqi He

Guanqi Hevor 11 Monaten

Dynamic feasible retargeting 🔥congrats jc-bao! 🦾！

Profilbild von Reza Sayar

Reza Sayarvor 11 Monaten

👏👏👏🙏🙏

Profilbild von Winston Gao

Winston Gaovor 11 Monaten

Impressive work! Hopefully one day we can retarget robot-spider motions using SPIDER!

Profilbild von Chaoyi Pan

Chaoyi Panvor 11 Monaten

haha, we will see ;)

Profilbild von Obligate Chronovore

Obligate Chronovorevor 11 Monaten

...this doesn't seem to work worth shit, why are you all spamming it so hard?

Profilbild von Siddharth Ancha

Siddharth Anchavor 11 Monaten

Really cool work!! 👏👏 Since you perform dynamics-level retargeting using sampling-based MPC that generates actions, I wonder if instead of "RL policy training" (Sec 4.2) you could just distill MPC into a policy using imitation learning / DAgger?

Ähnliche Videos

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

40,380 Aufrufe • vor 5 Monaten

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

Tairan He

97,008 Aufrufe • vor 2 Jahren

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ COLM @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,358 Aufrufe • vor 2 Jahren

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Yanjie Ze

88,323 Aufrufe • vor 1 Jahr

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,892 Aufrufe • vor 1 Jahr

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Irmak Guzey

65,141 Aufrufe • vor 2 Jahren

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Lerrel Pinto

84,611 Aufrufe • vor 3 Jahren

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

Mandi Zhao

122,225 Aufrufe • vor 1 Jahr

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 🎥 1/9

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 🎥 1/9

Zhen Wu

829,318 Aufrufe • vor 1 Jahr

Today we give Atlas dexterous hands, with the strength to take on real work, and the simplicity for mass manufacturing. Like the body, the hands are designed with high fidelity simulation in mind, to enable sim-to-real RL. If you give dexterous hands to a robot, make them do dexterous things. More thoughts here Blog: Video:

Today we give Atlas dexterous hands, with the strength to take on real work, and the simplicity for mass manufacturing. Like the body, the hands are designed with high fidelity simulation in mind, to enable sim-to-real RL. If you give dexterous hands to a robot, make them do dexterous things. More thoughts here Blog: Video:

Alberto Rodriguez

560,237 Aufrufe • vor 25 Tagen

How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.

How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.

Jianglong Ye

73,385 Aufrufe • vor 1 Jahr

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Lucy Shi

122,827 Aufrufe • vor 2 Jahren

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

Chen Wang

239,752 Aufrufe • vor 2 Jahren

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 Aufrufe • vor 1 Jahr

Mimic Robotics released a full-stack dexterity platform: the mimic hand M1 + umimic (U1) wearable. They keep human hand morphology consistent across the entire data-model-dexterous hand-manipulation pipeline--from video pre-training to wearable demos and robot deployment. This avoids cross-embodiment gaps and captures the essence of upper-limb intelligence. M1: Tendon-driven (15 active DoF), highly backdrivable ( 25kg grasp, industrial-grade, Swiss-made. U1 lets humans teach the robot naturally with their own hands--1:1 kinematics and sensing match, no latency or retargeting, enabling higher-quality and more scalable data. Looking forward to its performance at the customer site.

Mimic Robotics released a full-stack dexterity platform: the mimic hand M1 + umimic (U1) wearable. They keep human hand morphology consistent across the entire data-model-dexterous hand-manipulation pipeline--from video pre-training to wearable demos and robot deployment. This avoids cross-embodiment gaps and captures the essence of upper-limb intelligence. M1: Tendon-driven (15 active DoF), highly backdrivable ( 25kg grasp, industrial-grade, Swiss-made. U1 lets humans teach the robot naturally with their own hands--1:1 kinematics and sensing match, no latency or retargeting, enabling higher-quality and more scalable data. Looking forward to its performance at the customer site.

CyberRobo

26,160 Aufrufe • vor 3 Monaten

🎓 Excited to defend my PhD thesis “Learning Universal Humanoid Control” at CMU this Friday! From scalable motion imitators to visual dexterous whole-body policies — it’s been a wild ride 🤖✨ 📅 April 25, 2025 📍 CMU RI & online 🔗

🎓 Excited to defend my PhD thesis “Learning Universal Humanoid Control” at CMU this Friday! From scalable motion imitators to visual dexterous whole-body policies — it’s been a wild ride 🤖✨ 📅 April 25, 2025 📍 CMU RI & online 🔗

Zhengyi “Zen” Luo

49,544 Aufrufe • vor 1 Jahr

Retargeting test in @Cascadeur_soft Feat. Retargeting + Animation Unbaking + Fulcrum Cleaning + Physics (a little bit) All done in #Cascadeur #3danimation #gamedev #indiedev

Retargeting test in @Cascadeur_soft Feat. Retargeting + Animation Unbaking + Fulcrum Cleaning + Physics (a little bit) All done in #Cascadeur #3danimation #gamedev #indiedev

Eugene Dyabin

23,273 Aufrufe • vor 2 Jahren

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

The Humanoid Hub

106,447 Aufrufe • vor 1 Jahr

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

Chelsea Finn

264,494 Aufrufe • vor 3 Jahren

Video2Humanoid. Still trouble with bad retargeted humanoid motions? Humanoids now are Easy to track any video using our new Neural Motion Retargeting (NMR) Method. Optimization-based motion retargeting methods like IK and GMR are solving a non-convex problem frame by frame, which makes them sensitive to initialization, hard to tune, and prone to poor local minima. The result is familiar: joint discontinuities, self-collisions, and unstable foot contact. Our solution is simple but effective : instead of optimizing each frame independently, learn the mapping between human motion and robot motion as a distribution via networks. 📊Results on Unitree G1 - 0 joint jumps - 54% fewer self-collision frames - 61% fewer joint limit violations - Faster convergence for downstream control policy training NMR turns motion retargeting from a fragile optimization problem into a learned, scalable pipeline for more stable humanoid motion. please visit for more visualization results. opensource:

Video2Humanoid. Still trouble with bad retargeted humanoid motions? Humanoids now are Easy to track any video using our new Neural Motion Retargeting (NMR) Method. Optimization-based motion retargeting methods like IK and GMR are solving a non-convex problem frame by frame, which makes them sensitive to initialization, hard to tune, and prone to poor local minima. The result is familiar: joint discontinuities, self-collisions, and unstable foot contact. Our solution is simple but effective : instead of optimizing each frame independently, learn the mapping between human motion and robot motion as a distribution via networks. 📊Results on Unitree G1 - 0 joint jumps - 54% fewer self-collision frames - 61% fewer joint limit violations - Faster convergence for downstream control policy training NMR turns motion retargeting from a fragile optimization problem into a learned, scalable pipeline for more stable humanoid motion. please visit for more visualization results. opensource:

Xiao-Xiao Long

24,245 Aufrufe • vor 6 Monaten