Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

We present HDMI, a simple and general framework for learning whole-body interaction skills directly from human videos — no manual reward engineering, no task-specific pipelines. 🤖 67 door traversals, 6 real-world tasks, 14 in simulation. 🔗

Haoyang Weng

1,046 subscribers

135,400 Aufrufe • vor 1 Jahr •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

31 Kommentare

Profilbild von Haoyang Weng

Haoyang Wengvor 1 Jahr

How it works: 1️⃣ Extract human & object motion from monocular RGB videos 2️⃣ Train RL policies with: • unified object representation • residual action space • interaction reward 3️⃣ Deploy zero-shot to real humanoids

Profilbild von Haoyang Weng

Haoyang Wengvor 1 Jahr

HDMI is capable of: ✅ 67 consecutive door runs (34 minutes!) ✅ Box loco-manipulation with whole-body coordination ✅ Complex multi-stage behaviors like Truman’s Bow ⚡... all with a single set of rewards and observations!

Profilbild von PrismaX

PrismaXvor 1 Jahr

Why is it wearing boxing gloves?

Profilbild von Haoyang Weng

Haoyang Wengvor 1 Jahr

just for protecting the wrist motors. the hardware can be damaged if the robot falls.

Profilbild von R2rule

R2rulevor 1 Jahr

Nice. But why the trainer human needs to move like a robot? shouldn't it be the other way around?

Profilbild von Bill Chambers

Bill Chambersvor 1 Jahr

why does the robot have kneepads and ankle socks on?

Profilbild von Lyceum

Lyceumvor 10 Monaten

Wild how far video-based learning has come, how robust it is to messy real-world footage and occlusions.

Profilbild von Chris Paxton

Chris Paxtonvor 1 Jahr

Cool stuff

Profilbild von Allison Smith

Allison Smithvor 14 Tagen

There's already something called HDMI. Pick a different name.

Profilbild von Chaim Itself

Chaim Itselfvor 1 Jahr

HDMI? Surely you could have chosen an acronym that doesn't conflict with an existing standard name

Profilbild von Paleo-Reactionary Groyper

Paleo-Reactionary Groypervor 1 Jahr

@Indian_Bronson Hey that name is already taken

Profilbild von Christopher Cook

Christopher Cookvor 1 Jahr

Can you change the name to something unique, surely it deserves that.

Profilbild von Yitang Li

Yitang Livor 1 Jahr

Congrats Haoyang!

Profilbild von Haoyang Weng

Haoyang Wengvor 1 Jahr

Also congrats to you!

Profilbild von Xiatao Sun

Xiatao Sunvor 1 Jahr

Impressive work! Learning whole-body skills directly from monocular videos without manual reward engineering is exactly what the field needs.

Profilbild von Carlos DP 🤖🇺🇸

Carlos DP 🤖🇺🇸vor 1 Jahr

Very cool work!

Profilbild von Haoyang Weng

Haoyang Wengvor 1 Jahr

@the_carlosdp thank you so much!

Profilbild von CIX 🦾

CIX 🦾vor 1 Jahr

great work man

Profilbild von SoloTech

SoloTechvor 1 Jahr

This is a big step toward generalizable robotics.

Profilbild von Wealth Archives

Wealth Archivesvor 1 Jahr

@REALCULTNEWS china ai is better rn example

Profilbild von PeterSullivanish

PeterSullivanishvor 1 Jahr

Great work! 👏👏🎉

Profilbild von Samvel

Samvelvor 1 Jahr

Gumarth

Profilbild von Ruslan Sergeev

Ruslan Sergeevvor 11 Monaten

@ElijahGalahad Huge congratulations on the publication! It really is a great publication! Could you please explain the purpose of the Teacher and the Student policies in the repository? These are not quite straightforward and not described in the paper. Thanks!

Profilbild von Haoyang Weng

Haoyang Wengvor 11 Monaten

Thanks for your interest! ROA stands for regularized online adaptation from arxiv 2210.10044. It's for using more information during teacher training and distill them for student. It may speedup the entire training and help exploration, though direct train could also work.

Profilbild von Ruslan Sergeev

Ruslan Sergeevvor 11 Monaten

Thank you very much for the swift reply!

Profilbild von Samuel Friday

Samuel Fridayvor 1 Jahr

I think edge cases are the downside But this kind of data could also be mass collected and used for training

Profilbild von Oli

Olivor 1 Jahr

this is incredibly impressive the walking is perfect its really smooth it can use stairs move various objects this is a massive step for general robotics

Profilbild von Jude Onyenze

Jude Onyenzevor 1 Jahr

YOOOO, 3rd Person Video for Whole Body Humanoid Control!! Super cool, were you able to deploy for all tasks?

Profilbild von Haoyang Weng

Haoyang Wengvor 1 Jahr

I deployed 6 out of 14 tasks. mostly because of hardware limits, e.g. we cannot put mocap markers on a ball/we do not have a foldchair that have its back fixed.

Profilbild von Jude Onyenze

Jude Onyenzevor 1 Jahr

It still amazing results, maybe the hardware limits can be fixed with new models

Profilbild von MyDick

MyDickvor 1 Jahr

Pickup stuff carry wounded and disabled people, rescue trapped animals, do a personal security to grab weapons on trains and shield the innocent. That should be the main focus not that karate bullshit the other company was promoting

Ähnliche Videos

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 Aufrufe • vor 1 Jahr

🚀 Can we make a humanoid move like Cristiano Ronaldo, LeBron James and Kobe Byrant? YES! 🤖 Introducing ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills Website: Code:

🚀 Can we make a humanoid move like Cristiano Ronaldo, LeBron James and Kobe Byrant? YES! 🤖 Introducing ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills Website: Code:

Tairan He

259,205 Aufrufe • vor 1 Jahr

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

Jianfei Yang

18,740 Aufrufe • vor 1 Monat

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

The Humanoid Hub

31,445 Aufrufe • vor 10 Monaten

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

Chenhao Li

22,062 Aufrufe • vor 3 Monaten

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

Ashish Kapoor

118,467 Aufrufe • vor 1 Jahr

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

The Humanoid Hub

20,358 Aufrufe • vor 7 Monaten

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,294 Aufrufe • vor 2 Jahren

In the last month, we’ve been building an open-source framework for robot learning and sim-to-real transfer, made for RL whole-body control from simple walking to complex human imitation Check out the details on HN: Get started in 5 minutes ⬇️

In the last month, we’ve been building an open-source framework for robot learning and sim-to-real transfer, made for RL whole-body control from simple walking to complex human imitation Check out the details on HN: Get started in 5 minutes ⬇️

K-Scale Labs

75,983 Aufrufe • vor 1 Jahr

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,345 Aufrufe • vor 2 Jahren

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

Galbot

46,120 Aufrufe • vor 6 Monaten

Boston Dynamics has demonstrated its new Atlas humanoid robot lifting and carrying a 50-pound fridge. The robot uses its whole body to balance and move the heavy object in a natural, human-like way. Before doing it in the real world, Atlas practiced the task for millions of hours in a virtual simulation.

Boston Dynamics has demonstrated its new Atlas humanoid robot lifting and carrying a 50-pound fridge. The robot uses its whole body to balance and move the heavy object in a natural, human-like way. Before doing it in the real world, Atlas practiced the task for millions of hours in a virtual simulation.

Space and Technology

180,862 Aufrufe • vor 3 Monaten

What if we're approaching robotic manipulation all wrong? Ilya Sutskever makes a fascinating point here: Current approaches throw massive amounts of data at the problem — millions of simulation steps, huge compute. And still, robots can't match human dexterity. Real-world learning of new skills? "Very out of reach." Meanwhile, humans pick up manipulation tasks almost instantly. Our dexterity is unmatched. And we do it with remarkable robustness — no reward shaping, no curriculum design, no training instability. Why the gap? Evolution. 500M years of optimisation for locomotion, vision, and manipulation. For these ancient sensorimotor tasks, maybe brute-force data isn't the answer. Maybe we should be copying the biological algorithms that already solved it. CC: Dwarkesh Patel

What if we're approaching robotic manipulation all wrong? Ilya Sutskever makes a fascinating point here: Current approaches throw massive amounts of data at the problem — millions of simulation steps, huge compute. And still, robots can't match human dexterity. Real-world learning of new skills? "Very out of reach." Meanwhile, humans pick up manipulation tasks almost instantly. Our dexterity is unmatched. And we do it with remarkable robustness — no reward shaping, no curriculum design, no training instability. Why the gap? Evolution. 500M years of optimisation for locomotion, vision, and manipulation. For these ancient sensorimotor tasks, maybe brute-force data isn't the answer. Maybe we should be copying the biological algorithms that already solved it. CC: Dwarkesh Patel

Jack 🤖

31,262 Aufrufe • vor 10 Monaten

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 Aufrufe • vor 1 Jahr

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Siyuan Huang

22,019 Aufrufe • vor 11 Monaten

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Abhishek Gupta

12,086 Aufrufe • vor 1 Jahr