正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

We present HDMI, a simple and general framework for learning whole-body interaction skills directly from human videos — no manual reward engineering, no task-specific pipelines. 🤖 67 door traversals, 6 real-world tasks, 14 in simulation. 🔗

Haoyang Weng

1,046 subscribers

135,400 次观看 • 1 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

31 条评论

Haoyang Weng 的头像

Haoyang Weng1 年前

How it works: 1️⃣ Extract human & object motion from monocular RGB videos 2️⃣ Train RL policies with: • unified object representation • residual action space • interaction reward 3️⃣ Deploy zero-shot to real humanoids

Haoyang Weng 的头像

Haoyang Weng1 年前

HDMI is capable of: ✅ 67 consecutive door runs (34 minutes!) ✅ Box loco-manipulation with whole-body coordination ✅ Complex multi-stage behaviors like Truman’s Bow ⚡... all with a single set of rewards and observations!

PrismaX 的头像

PrismaX1 年前

Why is it wearing boxing gloves?

Haoyang Weng 的头像

Haoyang Weng1 年前

just for protecting the wrist motors. the hardware can be damaged if the robot falls.

R2rule 的头像

R2rule1 年前

Nice. But why the trainer human needs to move like a robot? shouldn't it be the other way around?

Bill Chambers 的头像

Bill Chambers1 年前

why does the robot have kneepads and ankle socks on?

Lyceum 的头像

Lyceum10 个月前

Wild how far video-based learning has come, how robust it is to messy real-world footage and occlusions.

Chris Paxton 的头像

Chris Paxton1 年前

Cool stuff

Allison Smith 的头像

Allison Smith14 天前

There's already something called HDMI. Pick a different name.

Chaim Itself 的头像

Chaim Itself1 年前

HDMI? Surely you could have chosen an acronym that doesn't conflict with an existing standard name

Paleo-Reactionary Groyper 的头像

Paleo-Reactionary Groyper1 年前

@Indian_Bronson Hey that name is already taken

Christopher Cook 的头像

Christopher Cook1 年前

Can you change the name to something unique, surely it deserves that.

Yitang Li 的头像

Yitang Li1 年前

Congrats Haoyang!

Haoyang Weng 的头像

Haoyang Weng1 年前

Also congrats to you!

Xiatao Sun 的头像

Xiatao Sun1 年前

Impressive work! Learning whole-body skills directly from monocular videos without manual reward engineering is exactly what the field needs.

Carlos DP 🤖🇺🇸 的头像

Carlos DP 🤖🇺🇸1 年前

Very cool work!

Haoyang Weng 的头像

Haoyang Weng1 年前

@the_carlosdp thank you so much!

CIX 🦾 的头像

CIX 🦾1 年前

great work man

SoloTech 的头像

SoloTech1 年前

This is a big step toward generalizable robotics.

Wealth Archives 的头像

Wealth Archives1 年前

@REALCULTNEWS china ai is better rn example

PeterSullivanish 的头像

PeterSullivanish1 年前

Great work! 👏👏🎉

Samvel 的头像

Samvel1 年前

Gumarth

Ruslan Sergeev 的头像

Ruslan Sergeev11 个月前

@ElijahGalahad Huge congratulations on the publication! It really is a great publication! Could you please explain the purpose of the Teacher and the Student policies in the repository? These are not quite straightforward and not described in the paper. Thanks!

Haoyang Weng 的头像

Haoyang Weng11 个月前

Thanks for your interest! ROA stands for regularized online adaptation from arxiv 2210.10044. It's for using more information during teacher training and distill them for student. It may speedup the entire training and help exploration, though direct train could also work.

Ruslan Sergeev 的头像

Ruslan Sergeev11 个月前

Thank you very much for the swift reply!

Samuel Friday 的头像

Samuel Friday1 年前

I think edge cases are the downside But this kind of data could also be mass collected and used for training

Oli 的头像

Oli1 年前

this is incredibly impressive the walking is perfect its really smooth it can use stairs move various objects this is a massive step for general robotics

Jude Onyenze 的头像

Jude Onyenze1 年前

YOOOO, 3rd Person Video for Whole Body Humanoid Control!! Super cool, were you able to deploy for all tasks?

Haoyang Weng 的头像

Haoyang Weng1 年前

I deployed 6 out of 14 tasks. mostly because of hardware limits, e.g. we cannot put mocap markers on a ball/we do not have a foldchair that have its back fixed.

Jude Onyenze 的头像

Jude Onyenze1 年前

It still amazing results, maybe the hardware limits can be fixed with new models

MyDick 的头像

MyDick1 年前

Pickup stuff carry wounded and disabled people, rescue trapped animals, do a personal security to grab weapons on trains and shield the innocent. That should be the main focus not that karate bullshit the other company was promoting

相关视频

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 次观看 • 1 年前

🚀 Can we make a humanoid move like Cristiano Ronaldo, LeBron James and Kobe Byrant? YES! 🤖 Introducing ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills Website: Code:

🚀 Can we make a humanoid move like Cristiano Ronaldo, LeBron James and Kobe Byrant? YES! 🤖 Introducing ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills Website: Code:

Tairan He

259,205 次观看 • 1 年前

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

Jianfei Yang

18,740 次观看 • 1 个月前

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

The Humanoid Hub

31,445 次观看 • 10 个月前

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

Chenhao Li

22,062 次观看 • 3 个月前

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

Ashish Kapoor

118,467 次观看 • 1 年前

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

The Humanoid Hub

20,358 次观看 • 7 个月前

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,294 次观看 • 2 年前

In the last month, we’ve been building an open-source framework for robot learning and sim-to-real transfer, made for RL whole-body control from simple walking to complex human imitation Check out the details on HN: Get started in 5 minutes ⬇️

In the last month, we’ve been building an open-source framework for robot learning and sim-to-real transfer, made for RL whole-body control from simple walking to complex human imitation Check out the details on HN: Get started in 5 minutes ⬇️

K-Scale Labs

75,983 次观看 • 1 年前

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,345 次观看 • 2 年前

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

Galbot

46,120 次观看 • 6 个月前

Boston Dynamics has demonstrated its new Atlas humanoid robot lifting and carrying a 50-pound fridge. The robot uses its whole body to balance and move the heavy object in a natural, human-like way. Before doing it in the real world, Atlas practiced the task for millions of hours in a virtual simulation.

Boston Dynamics has demonstrated its new Atlas humanoid robot lifting and carrying a 50-pound fridge. The robot uses its whole body to balance and move the heavy object in a natural, human-like way. Before doing it in the real world, Atlas practiced the task for millions of hours in a virtual simulation.

Space and Technology

180,862 次观看 • 3 个月前

What if we're approaching robotic manipulation all wrong? Ilya Sutskever makes a fascinating point here: Current approaches throw massive amounts of data at the problem — millions of simulation steps, huge compute. And still, robots can't match human dexterity. Real-world learning of new skills? "Very out of reach." Meanwhile, humans pick up manipulation tasks almost instantly. Our dexterity is unmatched. And we do it with remarkable robustness — no reward shaping, no curriculum design, no training instability. Why the gap? Evolution. 500M years of optimisation for locomotion, vision, and manipulation. For these ancient sensorimotor tasks, maybe brute-force data isn't the answer. Maybe we should be copying the biological algorithms that already solved it. CC: Dwarkesh Patel

What if we're approaching robotic manipulation all wrong? Ilya Sutskever makes a fascinating point here: Current approaches throw massive amounts of data at the problem — millions of simulation steps, huge compute. And still, robots can't match human dexterity. Real-world learning of new skills? "Very out of reach." Meanwhile, humans pick up manipulation tasks almost instantly. Our dexterity is unmatched. And we do it with remarkable robustness — no reward shaping, no curriculum design, no training instability. Why the gap? Evolution. 500M years of optimisation for locomotion, vision, and manipulation. For these ancient sensorimotor tasks, maybe brute-force data isn't the answer. Maybe we should be copying the biological algorithms that already solved it. CC: Dwarkesh Patel

Jack 🤖

31,262 次观看 • 10 个月前

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 次观看 • 1 年前

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Siyuan Huang

22,019 次观看 • 11 个月前

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Abhishek Gupta

12,086 次观看 • 1 年前