Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introduce HumanPlus - Shadowing part Humanoids are born for using human data. We build a real-time shadowing system using a single RGB camera and a whole-body policy for cloning human motion. Examples: - boxing🥊 - playing the piano🎹/ping pong - tossing - typing Open-sourced!

Zipeng Fu

13,227 subscribers

230,204 views • 2 years ago •via X (Twitter)

Gaming Science & Technology Education

Anya Rossi• Live Now

Private livecam show

10 Comments

Zipeng Fu2 years ago

Which hardware platform should HumanPlus be embodied on? We build our own 33-DoF humanoid with two dexterous hands using components: - Inspire-Robots RH56DFX hands - @UnitreeRobotics H1 robot - @ROBOTIS Dynamixel motors - @Razer webcams We open-source our hardware design.

Zipeng Fu2 years ago

Naively copying joints from humans to humanoids does not work due to gravity and different actuations. We train a transformer-based whole-body RL policy in IsaacGym simulation with realistic physics using AMASS dataset containing 40 hours of human motion:

Zipeng Fu2 years ago

To retarget from humans to humanoids, we copy the corresponding Euler angles from SMPL-X to our humanoid model. We use open-sourced SOTA human pose and hand estimation methods (thanks!) - WHAM for body: - HaMeR for hands:

Zipeng Fu2 years ago

Compared with other teleoperation methods, shadowing - is affordable - requires only 1 human operator - avoids singularities - natively supports whole-body control

Zipeng Fu2 years ago

Shadowing is an efficient data collection pipeline. We then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously by imitating human skills.

Zipeng Fu2 years ago

This project is not possible without our team of experts, covering from computer graphics to robot learning to robot hardware: - co-leads: @qingqing_zhao_ @Qi_Wu577 - advisors: @chelseabfinn @GordonWetzstein project website: hardware: code:

Yuchen Jin2 years ago

how much can I buy it? I need ten of these buddies shadowing me

Michael Black2 years ago

Nice!

Boris Belousov2 years ago

Very impressive work!

Yi Li2 years ago

very impressive!

Related Videos

Introduce HumanPlus - Autonomous Skills part Humanoids are born for using human data. Imitating humans, our humanoid learns: - fold sweatshirts - unload objects from warehouse racks - diverse locomotion skills (squatting, jumping, standing) - greet another robot Open-sourced!

Introduce HumanPlus - Autonomous Skills part Humanoids are born for using human data. Imitating humans, our humanoid learns: - fold sweatshirts - unload objects from warehouse racks - diverse locomotion skills (squatting, jumping, standing) - greet another robot Open-sourced!

Zipeng Fu

158,237 views • 2 years ago

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

TWIST: a real-time teleoperation system for humanoid robots to mimic whole-body motions. Reference motion data is generated by retargeting human motion-capture data to the robot. Then, the controller is trained in simulation using reinforcement learning and behavior cloning.

The Humanoid Hub

46,715 views • 1 year ago

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗:

Tairan He

96,738 views • 2 years ago

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,305 views • 2 years ago

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Siyuan Huang

21,932 views • 9 months ago

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 views • 10 months ago

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

Zhengyi “Zen” Luo

62,208 views • 8 months ago

Introduce OmniH2O, a learning-based system for whole-body humanoid teleop and autonomy: 🦾Robust loco-mani policy 🦸Universal teleop interface: VR, verbal, RGB 🧠Autonomy via ShitposterAI or imitation 🔗Release the first whole-body humanoid dataset

Introduce OmniH2O, a learning-based system for whole-body humanoid teleop and autonomy: 🦾Robust loco-mani policy 🦸Universal teleop interface: VR, verbal, RGB 🧠Autonomy via ShitposterAI or imitation 🔗Release the first whole-body humanoid dataset

Tairan He

130,032 views • 2 years ago

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

The Humanoid Hub

25,125 views • 8 months ago

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

Galbot

45,784 views • 4 months ago

With Helix 02, Figure has introduced a new foundational layer for whole-body control that replaces 109k lines of hand-engineered C++ with a single neural prior for stable, natural motion. - Learned from 1,000+ hours of human motion data and sim-to-real RL across 200,000+ parallel environments. - A 10M-parameter neural network that takes full-body joint state and base motion as input and outputs joint-level actuator commands at 1 kHz (1,000 times a second).

With Helix 02, Figure has introduced a new foundational layer for whole-body control that replaces 109k lines of hand-engineered C++ with a single neural prior for stable, natural motion. - Learned from 1,000+ hours of human motion data and sim-to-real RL across 200,000+ parallel environments. - A 10M-parameter neural network that takes full-body joint state and base motion as input and outputs joint-level actuator commands at 1 kHz (1,000 times a second).

The Humanoid Hub

28,004 views • 6 months ago

All of these articles on the body area networks and they never bothered to inform us they were using our body are network for the new E health system?? Weird huh? Look at all those diagrams with a human body in them

All of these articles on the body area networks and they never bothered to inform us they were using our body are network for the new E health system?? Weird huh? Look at all those diagrams with a human body in them

Dawn

16,454 views • 1 year ago

X-Humanoid unveiled TG-VLA, which it calls the world's first whole-body VLA framework for full-size humanoid robots. The goal: move humanoids past "mobile dual-arm systems" to human-like, whole-body coordinated operation. Built on three technologies: - HEX: an open-source full-size humanoid VLA that learns across embodiments and coordinates the whole body. Claimed SOTA on real-world whole-body manipulation - HAF-VLA: hierarchical action flow that decomposes whole-body motion into subtasks. Best or joint-best on 6 benchmarks - DSRL-DCT: online RL for high-DoF humanoids in a compressed latent space. 100% success on mobile manipulation HEX is open-source.

X-Humanoid unveiled TG-VLA, which it calls the world's first whole-body VLA framework for full-size humanoid robots. The goal: move humanoids past "mobile dual-arm systems" to human-like, whole-body coordinated operation. Built on three technologies: - HEX: an open-source full-size humanoid VLA that learns across embodiments and coordinates the whole body. Claimed SOTA on real-world whole-body manipulation - HAF-VLA: hierarchical action flow that decomposes whole-body motion into subtasks. Best or joint-best on 6 benchmarks - DSRL-DCT: online RL for high-DoF humanoids in a compressed latent space. 100% success on mobile manipulation HEX is open-source.

The Humanoid Hub

19,563 views • 28 days ago

THE ALPHA YOU WERE LOOKING FOR! Perle Labs a crypto project built on Solana with the aim to help and train AI models using real human knowledge and data. Perle Labs creates a system where experts do AI tasks, all on chain & transparent! Join here :

THE ALPHA YOU WERE LOOKING FOR! Perle Labs a crypto project built on Solana with the aim to help and train AI models using real human knowledge and data. Perle Labs creates a system where experts do AI tasks, all on chain & transparent! Join here :

ROSAAA

18,536 views • 5 months ago

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Excited to share our latest work on 🎧spatial audio-driven human motion generation. We aim to tackle a largely underexplored yet important problem of enabling virtual humans to move naturally in response to spatial audio—capturing not just what is heard, but also where the sound is coming from. To this end, we introduce the Spatial Audio-Driven Human Motion (SAM) dataset—the first comprehensive dataset featuring paired high-quality human motion and spatial audio recordings. For benchmarking, we develop a generative framework for human MOtion generation driven by SPAtial audio, termed MOSPA, which learns to synthesize realistic and diverse human motions conditioned on spatial audio input. We hope this research could provide a foundation for future research in spatial perception, virtual characters, and embodied AI. The dataset and model will be open-sourced soon. A big thank you to our intern, Shuyang Xu, for the wonderful collaboration! Congratulations, Shuyang! Project page: Paper: Video: #Animation #CG #CV #AIGC #DL #Deeplearning #Motion #Graphics #AI #GenerativeAI

Zhiyang (Frank) Dou

14,610 views • 1 year ago

We are aware of the immense value of data in an AI powered world. Assam State Data Policy 2026 strikes a fine balance between privacy and using anonymized data for better policy outcomes, informed choices and drives citizen centric governance.

We are aware of the immense value of data in an AI powered world. Assam State Data Policy 2026 strikes a fine balance between privacy and using anonymized data for better policy outcomes, informed choices and drives citizen centric governance.

Himanta Biswa Sarma

30,191 views • 27 days ago

A guy engineered an AI-powered pigeon defense system for his balcony Using a camera, AI detection, and servo-mounted targeting, the system identifies pigeons in real time and automatically sprays them with water 📹 muxamilian

A guy engineered an AI-powered pigeon defense system for his balcony Using a camera, AI detection, and servo-mounted targeting, the system identifies pigeons in real time and automatically sprays them with water 📹 muxamilian

Science girl

502,174 views • 2 months ago

We just open-sourced 3DREAL ! A new render-to-real IC-LoRA for LTX-2.3 from fal: turn any 3D / CG / game render into high 3d render, photorealistic, film-quality video, keeping your exact composition and camera move using the first frame.

We just open-sourced 3DREAL ! A new render-to-real IC-LoRA for LTX-2.3 from fal: turn any 3D / CG / game render into high 3d render, photorealistic, film-quality video, keeping your exact composition and camera move using the first frame.

fal

213,954 views • 1 month ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 views • 3 years ago

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 views • 3 years ago