Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

🕸️ Introducing SPIDER — Scalable Physics-Informed Dexterous Retargeting! A dynamically feasible, cross-embodiment retargeting framework for BOTH humanoids 🤖 and dexterous hands ✋. From human motion → sim → real robots, at scale. 🔗 Website: 🧵 1/n

Chaoyi Pan

1,105 subscribers

66,077 görüntüleme • 11 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

21 Yorum

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

🧵 2/n Retargeting human motion is hard: embodiment gap, missing forces, noisy data. IK fails ❌, RL is costly ❌. SPIDER converts kinematic human demos into dynamically feasible robot trajectories using physics-informed sampling + virtual contact guidance. 👇 Pipeline

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

🧵 3/n Because SPIDER is grounded with physics, the generated motions are directly executable on real hardware—no tuning. Zero robot demos. No RL training. Just human → physics → real robot.

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

🧵 4/n SPIDER supports diverse embodiments: 🤖 humanoid + ✋ dexterous hand 📦 2.4M robot-feasible frames 🤖 9 embodiments 🧊 103 objects ⏱️ 800 hrs of data

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

🧵 5/n SPIDER is also a data engine: We vary object size, states, and scenes to produce massive, physics-valid training data.

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

🧵 6/n For more details, please refer to: 🔗 Website: 📄 Paper: Huge thanks to the team at FAIR + CMU! We hope SPIDER pushes scalable, physics-grounded retargeting forward 🕷️🤖

C. Zhang profil fotoğrafı

C. Zhang11 ay önce

jc-bao🦾

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

@ChongZitaZhang Thanks Chong!

Jingyun Yang profil fotoğrafı

Jingyun Yang11 ay önce

Amazing work. Congrats @ChaoyiPan!

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

Thanks, Jingyun!

Reza Sayar profil fotoğrafı

Reza Sayar11 ay önce

@pablovelagomez1 👀

Weishuai Zeng profil fotoğrafı

Weishuai Zeng11 ay önce

Really impressive! When kinematic-based retargeter fails to preserve necessary physical information(like contact during interaction) and neural-based retargeter is hard to directly train, such physics-aware retarger becomes as the best option. Looking forward to the release!

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

Thanks Weishuai!

Wenli Xiao profil fotoğrafı

Wenli Xiao11 ay önce

Very impressive work, congrats jc-bao!

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

Thank you Wenli!

Shucheng Kang profil fotoğrafı

Shucheng Kang11 ay önce

Congratulations!

Guanqi He profil fotoğrafı

Guanqi He11 ay önce

Dynamic feasible retargeting 🔥congrats jc-bao! 🦾！

Reza Sayar profil fotoğrafı

Reza Sayar11 ay önce

👏👏👏🙏🙏

Winston Gao profil fotoğrafı

Winston Gao11 ay önce

Impressive work! Hopefully one day we can retarget robot-spider motions using SPIDER!

Chaoyi Pan profil fotoğrafı

Chaoyi Pan11 ay önce

haha, we will see ;)

Obligate Chronovore profil fotoğrafı

Obligate Chronovore11 ay önce

...this doesn't seem to work worth shit, why are you all spamming it so hard?

Siddharth Ancha profil fotoğrafı

Siddharth Ancha11 ay önce

Really cool work!! 👏👏 Since you perform dynamics-level retargeting using sampling-based MPC that generates actions, I wonder if instead of "RL policy training" (Sec 4.2) you could just distill MPC into a policy using imitation learning / DAgger?

Benzer Videolar

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Advancing dexterous manipulation through scalable visual sim-to-real transfer. We are excited to share our RSS paper, “ViserDex: Visual Sim-to-Real for Robust Dexterous In-hand Reorientation.” 🌐 Project page: 1/N 🧵

Robotic Systems Lab

40,380 görüntüleme • 5 ay önce

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

Tairan He

97,008 görüntüleme • 2 yıl önce

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ COLM @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,358 görüntüleme • 2 yıl önce

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Yanjie Ze

88,323 görüntüleme • 1 yıl önce

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks robust generalization and high performance without human demonstration :D

Toru

49,892 görüntüleme • 1 yıl önce

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Learning dexterous policies from human videos is challenging due to differences between human and robot hands. We present HuDOR, a method that learns dexterous policies within the robot's physical constraints using just one human video and an hour of online interactions! [1/n]

Irmak Guzey

65,141 görüntüleme • 2 yıl önce

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. 🧵👇

Lerrel Pinto

84,611 görüntüleme • 3 yıl önce

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

Mandi Zhao

122,225 görüntüleme • 1 yıl önce

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 🎥 1/9

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 🎥 1/9

Zhen Wu

829,318 görüntüleme • 1 yıl önce

Today we give Atlas dexterous hands, with the strength to take on real work, and the simplicity for mass manufacturing. Like the body, the hands are designed with high fidelity simulation in mind, to enable sim-to-real RL. If you give dexterous hands to a robot, make them do dexterous things. More thoughts here Blog: Video:

Today we give Atlas dexterous hands, with the strength to take on real work, and the simplicity for mass manufacturing. Like the body, the hands are designed with high fidelity simulation in mind, to enable sim-to-real RL. If you give dexterous hands to a robot, make them do dexterous things. More thoughts here Blog: Video:

Alberto Rodriguez

560,237 görüntüleme • 25 gün önce

How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.

How to generate billion-scale manipulation demonstrations easily? Let us leverage generative models! 🤖✨ We introduce Dex1B, a framework that generates 1 BILLION diverse dexterous hand demonstrations for both grasping 🖐️and articulation 💻 tasks using a simple C-VAE model.

Jianglong Ye

73,385 görüntüleme • 1 yıl önce

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Lucy Shi

122,827 görüntüleme • 2 yıl önce

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

Chen Wang

239,752 görüntüleme • 2 yıl önce

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 görüntüleme • 1 yıl önce

Mimic Robotics released a full-stack dexterity platform: the mimic hand M1 + umimic (U1) wearable. They keep human hand morphology consistent across the entire data-model-dexterous hand-manipulation pipeline--from video pre-training to wearable demos and robot deployment. This avoids cross-embodiment gaps and captures the essence of upper-limb intelligence. M1: Tendon-driven (15 active DoF), highly backdrivable ( 25kg grasp, industrial-grade, Swiss-made. U1 lets humans teach the robot naturally with their own hands--1:1 kinematics and sensing match, no latency or retargeting, enabling higher-quality and more scalable data. Looking forward to its performance at the customer site.

Mimic Robotics released a full-stack dexterity platform: the mimic hand M1 + umimic (U1) wearable. They keep human hand morphology consistent across the entire data-model-dexterous hand-manipulation pipeline--from video pre-training to wearable demos and robot deployment. This avoids cross-embodiment gaps and captures the essence of upper-limb intelligence. M1: Tendon-driven (15 active DoF), highly backdrivable ( 25kg grasp, industrial-grade, Swiss-made. U1 lets humans teach the robot naturally with their own hands--1:1 kinematics and sensing match, no latency or retargeting, enabling higher-quality and more scalable data. Looking forward to its performance at the customer site.

CyberRobo

26,160 görüntüleme • 3 ay önce

🎓 Excited to defend my PhD thesis “Learning Universal Humanoid Control” at CMU this Friday! From scalable motion imitators to visual dexterous whole-body policies — it’s been a wild ride 🤖✨ 📅 April 25, 2025 📍 CMU RI & online 🔗

🎓 Excited to defend my PhD thesis “Learning Universal Humanoid Control” at CMU this Friday! From scalable motion imitators to visual dexterous whole-body policies — it’s been a wild ride 🤖✨ 📅 April 25, 2025 📍 CMU RI & online 🔗

Zhengyi “Zen” Luo

49,544 görüntüleme • 1 yıl önce

Retargeting test in @Cascadeur_soft Feat. Retargeting + Animation Unbaking + Fulcrum Cleaning + Physics (a little bit) All done in #Cascadeur #3danimation #gamedev #indiedev

Retargeting test in @Cascadeur_soft Feat. Retargeting + Animation Unbaking + Fulcrum Cleaning + Physics (a little bit) All done in #Cascadeur #3danimation #gamedev #indiedev

Eugene Dyabin

23,273 görüntüleme • 2 yıl önce

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

Amazon is training humanoids to move boxes. Makes sense! OmniRetarget is a data generation engine that enables complex loco-manipulation for humanoids. It uses offline retargeting from human MoCap datasets and augments data from single demos to produce 8 hours of trajectories that train RL policies in simulation, transferred zero-shot to real robots. Amazon FAR is the primary affiliation for the paper:

The Humanoid Hub

106,447 görüntüleme • 1 yıl önce

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

Chelsea Finn

264,494 görüntüleme • 3 yıl önce

Video2Humanoid. Still trouble with bad retargeted humanoid motions? Humanoids now are Easy to track any video using our new Neural Motion Retargeting (NMR) Method. Optimization-based motion retargeting methods like IK and GMR are solving a non-convex problem frame by frame, which makes them sensitive to initialization, hard to tune, and prone to poor local minima. The result is familiar: joint discontinuities, self-collisions, and unstable foot contact. Our solution is simple but effective : instead of optimizing each frame independently, learn the mapping between human motion and robot motion as a distribution via networks. 📊Results on Unitree G1 - 0 joint jumps - 54% fewer self-collision frames - 61% fewer joint limit violations - Faster convergence for downstream control policy training NMR turns motion retargeting from a fragile optimization problem into a learned, scalable pipeline for more stable humanoid motion. please visit for more visualization results. opensource:

Video2Humanoid. Still trouble with bad retargeted humanoid motions? Humanoids now are Easy to track any video using our new Neural Motion Retargeting (NMR) Method. Optimization-based motion retargeting methods like IK and GMR are solving a non-convex problem frame by frame, which makes them sensitive to initialization, hard to tune, and prone to poor local minima. The result is familiar: joint discontinuities, self-collisions, and unstable foot contact. Our solution is simple but effective : instead of optimizing each frame independently, learn the mapping between human motion and robot motion as a distribution via networks. 📊Results on Unitree G1 - 0 joint jumps - 54% fewer self-collision frames - 61% fewer joint limit violations - Faster convergence for downstream control policy training NMR turns motion retargeting from a fragile optimization problem into a learned, scalable pipeline for more stable humanoid motion. please visit for more visualization results. opensource:

Xiao-Xiao Long

24,245 görüntüleme • 6 ay önce