Loading video...

Video Failed to Load

Go Home

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TheHumanoidHub's profile picture

The Humanoid Hub

103,814 subscribers

46,715 views • 1 year ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊
0:32

Excited to open-source GMR: General Motion Retargeting. Real-time human-to-humanoid retargeting on your laptop. Supports diverse motion formats & robots. Unlock whole-body humanoid teleoperation (e.g., TWIST). video with 🔊

Yanjie Ze

88,323 views • 1 year ago

Full-body joint tracking + smart gloves capture every nuance of human motion, which is retargeted in real-time to a humanoid robot (here, Tesla Optimus) for smooth teleoperation. This is the power of inertial motion capture. More than control, mocap is the gold standard for collecting high-fidelity humanoid training data. Advanced AI algorithms transform raw accelerometer streams into precise 3D joint poses, turning human movement into lifelike robot action.
0:20

Full-body joint tracking + smart gloves capture every nuance of human motion, which is retargeted in real-time to a humanoid robot (here, Tesla Optimus) for smooth teleoperation. This is the power of inertial motion capture. More than control, mocap is the gold standard for collecting high-fidelity humanoid training data. Advanced AI algorithms transform raw accelerometer streams into precise 3D joint poses, turning human movement into lifelike robot action.

Reborn

95,507 views • 1 year ago

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.
1:11

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

The Humanoid Hub

25,125 views • 10 months ago

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid
1:25

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

Galbot

46,125 views • 6 months ago

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:26

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

59,835 views • 9 months ago

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video
0:24

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 views • 1 year ago

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi
1:15

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,345 views • 2 years ago

🚀 Galbot just released AstraBrain-WBC 0.5, the world’s first humanoid robot cerebellum foundation model. Trained on 20,000 hours of human motion data, it validates a Scaling Law for robot motion control and demonstrates zero-shot execution of unseen motions on real humanoids. The era of robot control foundation models is getting closer.
3:05

🚀 Galbot just released AstraBrain-WBC 0.5, the world’s first humanoid robot cerebellum foundation model. Trained on 20,000 hours of human motion data, it validates a Scaling Law for robot motion control and demonstrates zero-shot execution of unseen motions on real humanoids. The era of robot control foundation models is getting closer.

Eren Chen

10,270 views • 3 months ago

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.
2:37

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.

The Humanoid Hub

91,980 views • 1 year ago

Introduce HumanPlus - Shadowing part Humanoids are born for using human data. We build a real-time shadowing system using a single RGB camera and a whole-body policy for cloning human motion. Examples: - boxing🥊 - playing the piano🎹/ping pong - tossing - typing Open-sourced!
0:53

Introduce HumanPlus - Shadowing part Humanoids are born for using human data. We build a real-time shadowing system using a single RGB camera and a whole-body policy for cloning human motion. Examples: - boxing🥊 - playing the piano🎹/ping pong - tossing - typing Open-sourced!

Zipeng Fu

230,388 views • 2 years ago

Researchers developed a system called LATENT that teaches humanoid robots to play tennis using small pieces of human movement data. It learns basic motion fragments, then corrects and combines them to create natural movements and swings. The system was tested on the Unitree G1 humanoid robot, enabling it to hit balls and rally with human players.
0:22

Researchers developed a system called LATENT that teaches humanoid robots to play tennis using small pieces of human movement data. It learns basic motion fragments, then corrects and combines them to create natural movements and swings. The system was tested on the Unitree G1 humanoid robot, enabling it to hit balls and rally with human players.

Space and Technology

45,748 views • 5 months ago

Researchers trained a humanoid robot to play tennis using only 5 hours of motion capture data The robot can now sustain multi-shot rallies with human players, hitting balls traveling >15 m/s with a ~90% success rate AlphaGo for every sport is coming
1:31

Researchers trained a humanoid robot to play tennis using only 5 hours of motion capture data The robot can now sustain multi-shot rallies with human players, hitting balls traveling >15 m/s with a ~90% success rate AlphaGo for every sport is coming

Andrew Kang

1,900,189 views • 6 months ago

RobotMDM, by Disney Research, combines diffusion-based motion generation with RL to produce physics-aware humanoid motions from text prompts. Trained on human motion data with a reward surrogate for physical feasibility, it ensures realistic motions.
0:56

RobotMDM, by Disney Research, combines diffusion-based motion generation with RL to produce physics-aware humanoid motions from text prompts. Trained on human motion data with a reward surrogate for physical feasibility, it ensures realistic motions.

The Humanoid Hub

22,943 views • 2 years ago

UE5.6 real-time AI motion capture plugin with native MetaHuman support. Using only a webcam or uploaded video, the 1-billion-parameter model converts footage into real-time 3D motion data and enables automatic retargeting.
2:13

UE5.6 real-time AI motion capture plugin with native MetaHuman support. Using only a webcam or uploaded video, the 1-billion-parameter model converts footage into real-time 3D motion data and enables automatic retargeting.

CYANPUPPETS

22,421 views • 4 months ago

AI Sapiens is ROBOTIS’ upcoming open-source humanoid platform for whole-body motion, imitation learning, reinforcement learning, and Sim2Real deployment on real hardware. Dance demonstration by SNU’s SHAPE team.
0:30

AI Sapiens is ROBOTIS’ upcoming open-source humanoid platform for whole-body motion, imitation learning, reinforcement learning, and Sim2Real deployment on real hardware. Dance demonstration by SNU’s SHAPE team.

ROBOTIS AMERICA

26,450 views • 1 month ago

Reinforcement learning is used to speed the production of behavior for the Boston Dynamics Atlas humanoid robot. At the heart of the learning process is a physics-based simulator that generates training data for a variety of maneuvers.
1:10

Reinforcement learning is used to speed the production of behavior for the Boston Dynamics Atlas humanoid robot. At the heart of the learning process is a physics-based simulator that generates training data for a variety of maneuvers.

RAI Institute

76,683 views • 1 year ago

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.
7:12

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 views • 3 years ago