Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

We present HDMI, a simple and general framework for learning whole-body interaction skills directly from human videos — no manual reward engineering, no task-specific pipelines. 🤖 67 door traversals, 6 real-world tasks, 14 in simulation. 🔗

Haoyang Weng

1,046 subscribers

135,400 görüntüleme • 1 yıl önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

31 Yorum

Haoyang Weng profil fotoğrafı

Haoyang Weng1 yıl önce

How it works: 1️⃣ Extract human & object motion from monocular RGB videos 2️⃣ Train RL policies with: • unified object representation • residual action space • interaction reward 3️⃣ Deploy zero-shot to real humanoids

Haoyang Weng profil fotoğrafı

Haoyang Weng1 yıl önce

HDMI is capable of: ✅ 67 consecutive door runs (34 minutes!) ✅ Box loco-manipulation with whole-body coordination ✅ Complex multi-stage behaviors like Truman’s Bow ⚡... all with a single set of rewards and observations!

PrismaX profil fotoğrafı

PrismaX1 yıl önce

Why is it wearing boxing gloves?

Haoyang Weng profil fotoğrafı

Haoyang Weng1 yıl önce

just for protecting the wrist motors. the hardware can be damaged if the robot falls.

R2rule profil fotoğrafı

R2rule1 yıl önce

Nice. But why the trainer human needs to move like a robot? shouldn't it be the other way around?

Bill Chambers profil fotoğrafı

Bill Chambers1 yıl önce

why does the robot have kneepads and ankle socks on?

Lyceum profil fotoğrafı

Lyceum10 ay önce

Wild how far video-based learning has come, how robust it is to messy real-world footage and occlusions.

Chris Paxton profil fotoğrafı

Chris Paxton1 yıl önce

Cool stuff

Allison Smith profil fotoğrafı

Allison Smith14 gün önce

There's already something called HDMI. Pick a different name.

Chaim Itself profil fotoğrafı

Chaim Itself1 yıl önce

HDMI? Surely you could have chosen an acronym that doesn't conflict with an existing standard name

Paleo-Reactionary Groyper profil fotoğrafı

Paleo-Reactionary Groyper1 yıl önce

@Indian_Bronson Hey that name is already taken

Christopher Cook profil fotoğrafı

Christopher Cook1 yıl önce

Can you change the name to something unique, surely it deserves that.

Yitang Li profil fotoğrafı

Yitang Li1 yıl önce

Congrats Haoyang!

Haoyang Weng profil fotoğrafı

Haoyang Weng1 yıl önce

Also congrats to you!

Xiatao Sun profil fotoğrafı

Xiatao Sun1 yıl önce

Impressive work! Learning whole-body skills directly from monocular videos without manual reward engineering is exactly what the field needs.

Carlos DP 🤖🇺🇸 profil fotoğrafı

Carlos DP 🤖🇺🇸1 yıl önce

Very cool work!

Haoyang Weng profil fotoğrafı

Haoyang Weng1 yıl önce

@the_carlosdp thank you so much!

CIX 🦾 profil fotoğrafı

CIX 🦾1 yıl önce

great work man

SoloTech profil fotoğrafı

SoloTech1 yıl önce

This is a big step toward generalizable robotics.

Wealth Archives profil fotoğrafı

Wealth Archives1 yıl önce

@REALCULTNEWS china ai is better rn example

PeterSullivanish profil fotoğrafı

PeterSullivanish1 yıl önce

Great work! 👏👏🎉

Samvel profil fotoğrafı

Samvel1 yıl önce

Gumarth

Ruslan Sergeev profil fotoğrafı

Ruslan Sergeev11 ay önce

@ElijahGalahad Huge congratulations on the publication! It really is a great publication! Could you please explain the purpose of the Teacher and the Student policies in the repository? These are not quite straightforward and not described in the paper. Thanks!

Haoyang Weng profil fotoğrafı

Haoyang Weng11 ay önce

Thanks for your interest! ROA stands for regularized online adaptation from arxiv 2210.10044. It's for using more information during teacher training and distill them for student. It may speedup the entire training and help exploration, though direct train could also work.

Ruslan Sergeev profil fotoğrafı

Ruslan Sergeev11 ay önce

Thank you very much for the swift reply!

Samuel Friday profil fotoğrafı

Samuel Friday1 yıl önce

I think edge cases are the downside But this kind of data could also be mass collected and used for training

Oli profil fotoğrafı

Oli1 yıl önce

this is incredibly impressive the walking is perfect its really smooth it can use stairs move various objects this is a massive step for general robotics

Jude Onyenze profil fotoğrafı

Jude Onyenze1 yıl önce

YOOOO, 3rd Person Video for Whole Body Humanoid Control!! Super cool, were you able to deploy for all tasks?

Haoyang Weng profil fotoğrafı

Haoyang Weng1 yıl önce

I deployed 6 out of 14 tasks. mostly because of hardware limits, e.g. we cannot put mocap markers on a ball/we do not have a foldchair that have its back fixed.

Jude Onyenze profil fotoğrafı

Jude Onyenze1 yıl önce

It still amazing results, maybe the hardware limits can be fixed with new models

MyDick profil fotoğrafı

MyDick1 yıl önce

Pickup stuff carry wounded and disabled people, rescue trapped animals, do a personal security to grab weapons on trains and shield the innocent. That should be the main focus not that karate bullshit the other company was promoting

Benzer Videolar

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

HDMI (HumanoiD iMitation for Interaction) is a framework enabling humanoid robots to learn whole-body object interaction skills from monocular RGB human videos. It extracts and retargets human poses and object trajectories using GVHMR and LocoMujoco, building reference datasets with contact annotations. The data is used to train an RL policy via robot-object co-tracking. HDMI achieved 67 consecutive door traversals.

The Humanoid Hub

17,395 görüntüleme • 1 yıl önce

🚀 Can we make a humanoid move like Cristiano Ronaldo, LeBron James and Kobe Byrant? YES! 🤖 Introducing ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills Website: Code:

🚀 Can we make a humanoid move like Cristiano Ronaldo, LeBron James and Kobe Byrant? YES! 🤖 Introducing ASAP: Aligning Simulation and Real-World Physics for Learning Agile Humanoid Whole-Body Skills Website: Code:

Tairan He

259,205 görüntüleme • 1 yıl önce

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

🤖 Introducing ω-0 (OMEGA-0) — a whole-body World Action Model for humanoids. Can one model make a humanoid walk, manipulate, and coordinate its whole body simultaneously across many real-world tasks? ω-0 does exactly that. ⚡ One unified model for multi-task whole-body loco-manipulation 🧠 Predicts future visual latents while generating executable whole-body actions 🏠 Trained with ω-HOME: 40+ hours, 24 household tasks, 4,827 episodes 🔥 81.8% real-world success rate across 11 evaluation tasks No separate locomotion/manipulation policies. One model. Whole body. Multiple tasks. Humanoid intelligence should not be a collection of isolated skills. It should be a unified whole-body policy. 🔗 📜 One-take, uncut real-world demo👇

Jianfei Yang

18,740 görüntüleme • 1 ay önce

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

The Humanoid Hub

31,445 görüntüleme • 10 ay önce

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

🤷What if we want to learn from human data... without human data? In our work NIL (No-data Imitation Learning) #CVPR2026, we explore a simple but ambitious question: Can robots learn directly from AI-generated videos without any curated demonstration data? 🔗

Chenhao Li

22,062 görüntüleme • 3 ay önce

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

Ashish Kapoor

118,467 görüntüleme • 1 yıl önce

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

HumanX: a scalable framework that converts single monocular human videos into agile, generalizable interaction skills for humanoid robots without task-specific rewards. Core parts: - XGen: retargets human motion + synthesizes diverse physically plausible training data (via physics-driven object trajectories and augmentation). - XMimic: unified imitation pipeline for robust generalization. Shows promising results for both Blind (no external sensors) and MoCap modes.

The Humanoid Hub

20,358 görüntüleme • 7 ay önce

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Announcing RoboFuME🤖💨, a system for autonomous & efficient real-world robot learning that 1. pre-trains a VLM reward model and a multi-task RL policy from diverse off-the-shelf demo data; 2. runs RL fine-tuning online with the VLM reward model. 🔗 🧵↓

Jingyun Yang

88,294 görüntüleme • 2 yıl önce

In the last month, we’ve been building an open-source framework for robot learning and sim-to-real transfer, made for RL whole-body control from simple walking to complex human imitation Check out the details on HN: Get started in 5 minutes ⬇️

In the last month, we’ve been building an open-source framework for robot learning and sim-to-real transfer, made for RL whole-body control from simple walking to complex human imitation Check out the details on HN: Get started in 5 minutes ⬇️

K-Scale Labs

75,983 görüntüleme • 1 yıl önce

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,345 görüntüleme • 2 yıl önce

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

🎾Your humanoid tennis player is here!🤖 Introducing LATENT (Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data) — the world’s first real-time whole-body planning and control algorithm for athletic humanoid tennis. For the first time, a humanoid robot can sustain high-dynamic, long-horizon tennis rallies with millisecond-level reactions, precise ball striking, and natural whole-body motion. This marks a leap from mechanical motion imitation to intelligent, decision-driven athletic interaction. #Galbot #Robotics #Innovation #AI #Technology #Humanoid

Galbot

46,120 görüntüleme • 6 ay önce

Boston Dynamics has demonstrated its new Atlas humanoid robot lifting and carrying a 50-pound fridge. The robot uses its whole body to balance and move the heavy object in a natural, human-like way. Before doing it in the real world, Atlas practiced the task for millions of hours in a virtual simulation.

Boston Dynamics has demonstrated its new Atlas humanoid robot lifting and carrying a 50-pound fridge. The robot uses its whole body to balance and move the heavy object in a natural, human-like way. Before doing it in the real world, Atlas practiced the task for millions of hours in a virtual simulation.

Space and Technology

180,862 görüntüleme • 3 ay önce

What if we're approaching robotic manipulation all wrong? Ilya Sutskever makes a fascinating point here: Current approaches throw massive amounts of data at the problem — millions of simulation steps, huge compute. And still, robots can't match human dexterity. Real-world learning of new skills? "Very out of reach." Meanwhile, humans pick up manipulation tasks almost instantly. Our dexterity is unmatched. And we do it with remarkable robustness — no reward shaping, no curriculum design, no training instability. Why the gap? Evolution. 500M years of optimisation for locomotion, vision, and manipulation. For these ancient sensorimotor tasks, maybe brute-force data isn't the answer. Maybe we should be copying the biological algorithms that already solved it. CC: Dwarkesh Patel

What if we're approaching robotic manipulation all wrong? Ilya Sutskever makes a fascinating point here: Current approaches throw massive amounts of data at the problem — millions of simulation steps, huge compute. And still, robots can't match human dexterity. Real-world learning of new skills? "Very out of reach." Meanwhile, humans pick up manipulation tasks almost instantly. Our dexterity is unmatched. And we do it with remarkable robustness — no reward shaping, no curriculum design, no training instability. Why the gap? Evolution. 500M years of optimisation for locomotion, vision, and manipulation. For these ancient sensorimotor tasks, maybe brute-force data isn't the answer. Maybe we should be copying the biological algorithms that already solved it. CC: Dwarkesh Patel

Jack 🤖

31,262 görüntüleme • 10 ay önce

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 görüntüleme • 1 yıl önce

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Excited to introduce COLA: Learning Human–Humanoid Coordination for Collaborative Object Carrying 🤝🤖 COLA makes humanoids truly helpful in human collaboration — capable of carrying objects, pushing carts, or responding to human push commands. It provides a proprioception-only policy for compliant human–humanoid coordination across diverse movement patterns. The core idea is simple yet effective: 👉 Fine-tune a collaborative policy from a locomotion policy using a residual teacher. 👉 Train in simulation and distill to a real-world student policy for deployment. Paper: Project:

Siyuan Huang

22,019 görüntüleme • 11 ay önce

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Abhishek Gupta

12,086 görüntüleme • 1 yıl önce