Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

🕸️ Introducing SPIDER — Scalable Physics-Informed Dexterous Retargeting! A dynamically feasible, cross-embodiment retargeting framework for BOTH humanoids 🤖 and dexterous hands ✋. From human motion → sim → real robots, at scale. 🔗 Website: 🧵 1/n

Chaoyi Pan

1,105 subscribers

66,077 просмотров • 11 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 21

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

🧵 2/n Retargeting human motion is hard: embodiment gap, missing forces, noisy data. IK fails ❌, RL is costly ❌. SPIDER converts kinematic human demos into dynamically feasible robot trajectories using physics-informed sampling + virtual contact guidance. 👇 Pipeline

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

🧵 3/n Because SPIDER is grounded with physics, the generated motions are directly executable on real hardware—no tuning. Zero robot demos. No RL training. Just human → physics → real robot.

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

🧵 4/n SPIDER supports diverse embodiments: 🤖 humanoid + ✋ dexterous hand 📦 2.4M robot-feasible frames 🤖 9 embodiments 🧊 103 objects ⏱️ 800 hrs of data

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

🧵 5/n SPIDER is also a data engine: We vary object size, states, and scenes to produce massive, physics-valid training data.

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

🧵 6/n For more details, please refer to: 🔗 Website: 📄 Paper: Huge thanks to the team at FAIR + CMU! We hope SPIDER pushes scalable, physics-grounded retargeting forward 🕷️🤖

Фото профиля C. Zhang

C. Zhang11 месяцев назад

jc-bao🦾

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

@ChongZitaZhang Thanks Chong!

Фото профиля Jingyun Yang

Jingyun Yang11 месяцев назад

Amazing work. Congrats @ChaoyiPan!

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

Thanks, Jingyun!

Фото профиля Reza Sayar

Reza Sayar11 месяцев назад

@pablovelagomez1 👀

Фото профиля Weishuai Zeng

Weishuai Zeng11 месяцев назад

Really impressive! When kinematic-based retargeter fails to preserve necessary physical information(like contact during interaction) and neural-based retargeter is hard to directly train, such physics-aware retarger becomes as the best option. Looking forward to the release!

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

Thanks Weishuai!

Фото профиля Wenli Xiao

Wenli Xiao11 месяцев назад

Very impressive work, congrats jc-bao!

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

Thank you Wenli!

Фото профиля Shucheng Kang

Shucheng Kang11 месяцев назад

Congratulations!

Фото профиля Guanqi He

Guanqi He11 месяцев назад

Dynamic feasible retargeting 🔥congrats jc-bao! 🦾！

Фото профиля Reza Sayar

Reza Sayar11 месяцев назад

👏👏👏🙏🙏

Фото профиля Winston Gao

Winston Gao11 месяцев назад

Impressive work! Hopefully one day we can retarget robot-spider motions using SPIDER!

Фото профиля Chaoyi Pan

Chaoyi Pan11 месяцев назад

haha, we will see ;)

Фото профиля Obligate Chronovore

Obligate Chronovore11 месяцев назад

...this doesn't seem to work worth shit, why are you all spamming it so hard?

Фото профиля Siddharth Ancha

Siddharth Ancha11 месяцев назад

Really cool work!! 👏👏 Since you perform dynamics-level retargeting using sampling-based MPC that generates actions, I wonder if instead of "RL policy training" (Sec 4.2) you could just distill MPC into a policy using imitation learning / DAgger?

Похожие видео

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

40,380 просмотров • 5 месяцев назад

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

Tairan He

97,008 просмотров • 2 лет назад

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ COLM @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,358 просмотров • 2 лет назад

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Yanjie Ze

88,323 просмотров • 1 год назад

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,892 просмотров • 1 год назад

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Irmak Guzey

65,141 просмотров • 2 лет назад

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Lerrel Pinto

84,611 просмотров • 3 лет назад

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

Mandi Zhao

122,225 просмотров • 1 год назад

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 🎥 1/9

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 🎥 1/9

Zhen Wu

829,318 просмотров • 1 год назад

Today we give Atlas dexterous hands, with the strength to take on real work, and the simplicity for mass manufacturing. Like the body, the hands are designed with high fidelity simulation in mind, to enable sim-to-real RL. If you give dexterous hands to a robot, make them do dexterous things. More thoughts here Blog: Video:

Today we give Atlas dexterous hands, with the strength to take on real work, and the simplicity for mass manufacturing. Like the body, the hands are designed with high fidelity simulation in mind, to enable sim-to-real RL. If you give dexterous hands to a robot, make them do dexterous things. More thoughts here Blog: Video:

Alberto Rodriguez

560,237 просмотров • 25 дней назад

How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.

How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.

Jianglong Ye

73,385 просмотров • 1 год назад

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Lucy Shi

122,827 просмотров • 2 лет назад

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

Chen Wang

239,752 просмотров • 2 лет назад

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 просмотров • 1 год назад

Mimic Robotics released a full-stack dexterity platform: the mimic hand M1 + umimic (U1) wearable. They keep human hand morphology consistent across the entire data-model-dexterous hand-manipulation pipeline--from video pre-training to wearable demos and robot deployment. This avoids cross-embodiment gaps and captures the essence of upper-limb intelligence. M1: Tendon-driven (15 active DoF), highly backdrivable ( 25kg grasp, industrial-grade, Swiss-made. U1 lets humans teach the robot naturally with their own hands--1:1 kinematics and sensing match, no latency or retargeting, enabling higher-quality and more scalable data. Looking forward to its performance at the customer site.

Mimic Robotics released a full-stack dexterity platform: the mimic hand M1 + umimic (U1) wearable. They keep human hand morphology consistent across the entire data-model-dexterous hand-manipulation pipeline--from video pre-training to wearable demos and robot deployment. This avoids cross-embodiment gaps and captures the essence of upper-limb intelligence. M1: Tendon-driven (15 active DoF), highly backdrivable ( 25kg grasp, industrial-grade, Swiss-made. U1 lets humans teach the robot naturally with their own hands--1:1 kinematics and sensing match, no latency or retargeting, enabling higher-quality and more scalable data. Looking forward to its performance at the customer site.

CyberRobo

26,160 просмотров • 3 месяцев назад

🎓 Excited to defend my PhD thesis “Learning Universal Humanoid Control” at CMU this Friday! From scalable motion imitators to visual dexterous whole-body policies — it’s been a wild ride 🤖✨ 📅 April 25, 2025 📍 CMU RI & online 🔗

🎓 Excited to defend my PhD thesis “Learning Universal Humanoid Control” at CMU this Friday! From scalable motion imitators to visual dexterous whole-body policies — it’s been a wild ride 🤖✨ 📅 April 25, 2025 📍 CMU RI & online 🔗

Zhengyi “Zen” Luo

49,544 просмотров • 1 год назад

Retargeting test in @Cascadeur_soft Feat. Retargeting + Animation Unbaking + Fulcrum Cleaning + Physics (a little bit) All done in #Cascadeur #3danimation #gamedev #indiedev

Retargeting test in @Cascadeur_soft Feat. Retargeting + Animation Unbaking + Fulcrum Cleaning + Physics (a little bit) All done in #Cascadeur #3danimation #gamedev #indiedev

Eugene Dyabin

23,273 просмотров • 2 лет назад

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

The Humanoid Hub

106,447 просмотров • 1 год назад

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

Chelsea Finn

264,494 просмотров • 3 лет назад

Video2Humanoid. Still trouble with bad retargeted humanoid motions? Humanoids now are Easy to track any video using our new Neural Motion Retargeting (NMR) Method. Optimization-based motion retargeting methods like IK and GMR are solving a non-convex problem frame by frame, which makes them sensitive to initialization, hard to tune, and prone to poor local minima. The result is familiar: joint discontinuities, self-collisions, and unstable foot contact. Our solution is simple but effective : instead of optimizing each frame independently, learn the mapping between human motion and robot motion as a distribution via networks. 📊Results on Unitree G1 - 0 joint jumps - 54% fewer self-collision frames - 61% fewer joint limit violations - Faster convergence for downstream control policy training NMR turns motion retargeting from a fragile optimization problem into a learned, scalable pipeline for more stable humanoid motion. please visit for more visualization results. opensource:

Video2Humanoid. Still trouble with bad retargeted humanoid motions? Humanoids now are Easy to track any video using our new Neural Motion Retargeting (NMR) Method. Optimization-based motion retargeting methods like IK and GMR are solving a non-convex problem frame by frame, which makes them sensitive to initialization, hard to tune, and prone to poor local minima. The result is familiar: joint discontinuities, self-collisions, and unstable foot contact. Our solution is simple but effective : instead of optimizing each frame independently, learn the mapping between human motion and robot motion as a distribution via networks. 📊Results on Unitree G1 - 0 joint jumps - 54% fewer self-collision frames - 61% fewer joint limit violations - Faster convergence for downstream control policy training NMR turns motion retargeting from a fragile optimization problem into a learned, scalable pipeline for more stable humanoid motion. please visit for more visualization results. opensource:

Xiao-Xiao Long

24,245 просмотров • 6 месяцев назад