Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Our 6ft humanoid HumanPlus at Stanford can autonomously put on a Nike skateboard shoe, tie shoelaces, stand up and walk. Using two transformers & dual RGB vision, it integrates two recipes of general robotics end-to-end: - imitating humans in real world - large-scale RL in sim

Zipeng Fu

13,227 subscribers

246,224 views • 2 years ago •via X (Twitter)

Education News & Politics Science & Technology

Anya Rossi• Live Now

Private livecam show

10 Comments

Zipeng Fu2 years ago

More autonomous skills of HumanPlus can be found here:

Zipeng Fu2 years ago

Team: @zipengfu*, @qingqing_zhao_*, @Qi_Wu577*, @GordonWetzstein, @chelseabfinn project website: hardware: code:

Barrett2 years ago

What happens if I sneak behind it and tie the laces together when it’s not looking

Neil Lawrence2 years ago

But can it then say "I don't have to outrun the lion, I only have to outrun you."?

whistle2 years ago

why does it need shoes…

Hang Zhao2 years ago

This is truly impressive!

Zipeng Fu2 years ago

Thanks Hang!

stellarstrain2 years ago

Why 6 ft, 5.11 would have been better.

Ophelia2 years ago

Finally some sick sneakers on a bot. It's about time.

Happy2 years ago

wow😮

Related Videos

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵

Zero teleoperation. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. Introducing VIRAL: Visual Sim-to-Real at Scale. We achieved 54 autonomous cycles (walk, stand, place, pick, turn) using a simple recipe: 1. RL 2. Simulation 3. GPUs Website: Arxiv: Deep dive with me: 🧵

Tairan He

266,831 views • 9 months ago

Humanoid and Siemens successfully completed a POC testing humanoid robots in industrial logistics. HMND 01 robot autonomously performed a tote-to-conveyor destacking task during a two-week on-site deployment at the Siemens Electronics Factory in Erlangen. This is the first step in our broader partnership with Siemens as humanoid robots move closer to real-world deployment. Learn more here:

Humanoid and Siemens successfully completed a POC testing humanoid robots in industrial logistics. HMND 01 robot autonomously performed a tote-to-conveyor destacking task during a two-week on-site deployment at the Siemens Electronics Factory in Erlangen. This is the first step in our broader partnership with Siemens as humanoid robots move closer to real-world deployment. Learn more here:

Humanoid

22,914 views • 7 months ago

Two South Carolina teams going at it in the Senior League World Series and they didn't disappoint! #LLWS Top plays on both sides and a walk-off to end 🔥

Two South Carolina teams going at it in the Senior League World Series and they didn't disappoint! #LLWS Top plays on both sides and a walk-off to end 🔥

Little League

32,778 views • 3 years ago

After two years in stealth mode, we're thrilled to unveil Mentee Robotics and our humanoid robot, Mentee Robotics! With AI integration at every layer, from Sim2Real machine learning to NeRF-based algorithms and LLMs, we've achieved a complete end-to-end cycle for tasks. This is just the beginning and we can’t wait to share more

After two years in stealth mode, we're thrilled to unveil Mentee Robotics and our humanoid robot, Mentee Robotics! With AI integration at every layer, from Sim2Real machine learning to NeRF-based algorithms and LLMs, we've achieved a complete end-to-end cycle for tasks. This is just the beginning and we can’t wait to share more

Amnon Shashua

223,942 views • 2 years ago

Trained a humanoid entirely in a 3D scan of the office. Zero real-world fine-tuning. It just walked in and worked. RL needs hundreds of thousands of attempts, and real robots can't afford to crash. A misjudged gap or a glass door collision breaks hardware and costs hours resetting. So you train in a sim. But sim policies usually train on randomized, untextured geometry; depth is easy to fake. The robot learns structure, not the real world: no materials, no lighting, no idea what anything actually is. RGB cameras carry all of that but training RGB policies in generic fake worlds won’t generalize to the real world. Niantic Spatial 🌎 Scaniverse reconstructs your scan of the real deployment site. One 360° camera walkthrough → photorealistic 3D Gaussian splat at metric scale → collision mesh pulled from the same reconstruction, so vision and physics match exactly. Drops straight into NVIDIA Isaac Sim/Lab, no manual conversion. Flexion simulation-first approach then seamlessly enables the training of RGB-only nav policies inside that reconstruction. With added domain randomization + large image encoders for robustness, this deploys straight to hardware. No real-world fine-tuning. Deployment: months of on-site adaptation → days. Tune into the NVIDIA livestream on 12 August to hear how these companies are closing the sim2real gap: NVIDIA Robotics ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Trained a humanoid entirely in a 3D scan of the office. Zero real-world fine-tuning. It just walked in and worked. RL needs hundreds of thousands of attempts, and real robots can't afford to crash. A misjudged gap or a glass door collision breaks hardware and costs hours resetting. So you train in a sim. But sim policies usually train on randomized, untextured geometry; depth is easy to fake. The robot learns structure, not the real world: no materials, no lighting, no idea what anything actually is. RGB cameras carry all of that but training RGB policies in generic fake worlds won’t generalize to the real world. Niantic Spatial 🌎 Scaniverse reconstructs your scan of the real deployment site. One 360° camera walkthrough → photorealistic 3D Gaussian splat at metric scale → collision mesh pulled from the same reconstruction, so vision and physics match exactly. Drops straight into NVIDIA Isaac Sim/Lab, no manual conversion. Flexion simulation-first approach then seamlessly enables the training of RGB-only nav policies inside that reconstruction. With added domain randomization + large image encoders for robustness, this deploys straight to hardware. No real-world fine-tuning. Deployment: months of on-site adaptation → days. Tune into the NVIDIA livestream on 12 August to hear how these companies are closing the sim2real gap: NVIDIA Robotics ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

119,163 views • 26 days ago

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

Ashish Kapoor

118,406 views • 11 months ago

Israel-based Mentee Robotics has demonstrated a logistics workflow: two MenteeBot V3 humanoids work autonomously to pick and place totes. A Modular Agent System is preferred because it favors real-world robustness and lower compute needs over the End-to-End VLA model. Its architecture is composed of three components: - LLM Planner: Converts instructions into executable Robotic API Language code for reliable task decomposition and error handling. - Perception Stack: Uses pre-trained models (NeRF/3DGS, distilled vision) for scene understanding and navigation. - Control Policies: Reinforcement Learning (RL) models, trained at scale via Sim2Real, generate motor commands, enabling high-accuracy mobile manipulation. Crucially, the robot learns new tasks from a single demonstration in hours. Object tracking uses 3D geometry (STL/URDF) tracked in the video to define the RL reward function. Training is optimized using 'Automatic Curriculum Learning', which autonomously adjusts task difficulty based on robot performance, eliminating manual engineering. All computation runs onboard.

Israel-based Mentee Robotics has demonstrated a logistics workflow: two MenteeBot V3 humanoids work autonomously to pick and place totes. A Modular Agent System is preferred because it favors real-world robustness and lower compute needs over the End-to-End VLA model. Its architecture is composed of three components: - LLM Planner: Converts instructions into executable Robotic API Language code for reliable task decomposition and error handling. - Perception Stack: Uses pre-trained models (NeRF/3DGS, distilled vision) for scene understanding and navigation. - Control Policies: Reinforcement Learning (RL) models, trained at scale via Sim2Real, generate motor commands, enabling high-accuracy mobile manipulation. Crucially, the robot learns new tasks from a single demonstration in hours. Object tracking uses 3D geometry (STL/URDF) tracked in the video to define the RL reward function. Training is optimized using 'Automatic Curriculum Learning', which autonomously adjusts task difficulty based on robot performance, eliminating manual engineering. All computation runs onboard.

The Humanoid Hub

15,729 views • 9 months ago

🔋At CEEC Energy Storage Technology’s Intelligent Manufacturing Base, our industry-first automated large-module and full-pack assembly line integrates AGVs, robotics, machine vision, intelligent inspection, and MES systems for end-to-end smart control and seamless coordination. It enables fully automated production from prismatic aluminum-shell battery cells to modules and complete PACK systems. With a cell production rate of ≥16 PPM per line and an annual capacity of 6 GWh per line, we leverage large-scale, intelligent manufacturing to deliver reliable and efficient energy products to the market.✨🤩

🔋At CEEC Energy Storage Technology’s Intelligent Manufacturing Base, our industry-first automated large-module and full-pack assembly line integrates AGVs, robotics, machine vision, intelligent inspection, and MES systems for end-to-end smart control and seamless coordination. It enables fully automated production from prismatic aluminum-shell battery cells to modules and complete PACK systems. With a cell production rate of ≥16 PPM per line and an annual capacity of 6 GWh per line, we leverage large-scale, intelligent manufacturing to deliver reliable and efficient energy products to the market.✨🤩

Energy China

158,763 views • 2 months ago

Five years ago, OpenAI trained a 5-fingered humanoid hand to solve a Rubik’s Cube using RL and domain randomization, pushing the boundaries of sim-to-real for fine-motor tasks. It underscored the value of creating sufficiently complex simulated worlds in which robots can learn.

Five years ago, OpenAI trained a 5-fingered humanoid hand to solve a Rubik’s Cube using RL and domain randomization, pushing the boundaries of sim-to-real for fine-motor tasks. It underscored the value of creating sufficiently complex simulated worlds in which robots can learn.

The Humanoid Hub

66,761 views • 1 year ago

Introducing INTELLECT-3: Scaling RL to a 100B+ MoE model on our end-to-end stack Achieving state-of-the-art performance for its size across math, code and reasoning Built using the same tools we put in your hands, from environments & evals, RL frameworks, sandboxes & more

Introducing INTELLECT-3: Scaling RL to a 100B+ MoE model on our end-to-end stack Achieving state-of-the-art performance for its size across math, code and reasoning Built using the same tools we put in your hands, from environments & evals, RL frameworks, sandboxes & more

Prime Intellect

1,149,775 views • 9 months ago

🦾 Our robot hand can rotate objects over 6+ axes in the real-world! Introducing RotateIt (CoRL 2023), a Sim-to-Real policy that can rotate many objects over many axes, using vision and touch! Check it out: Paper: #CoRL2023

🦾 Our robot hand can rotate objects over 6+ axes in the real-world! Introducing RotateIt (CoRL 2023), a Sim-to-Real policy that can rotate many objects over many axes, using vision and touch! Check it out: Paper: #CoRL2023

Haozhi Qi

51,041 views • 3 years ago

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 views • 1 year ago

The Information: Amazon is gearing up to test humanoid robots that walk on two legs to deliver packages. The company has set up an indoor obstacle course to test robots from a variety of companies, including Unitree. Eventually, humanoid robots could ride in Amazon’s electric Rivian delivery vans and autonomously deliver packages.

The Information: Amazon is gearing up to test humanoid robots that walk on two legs to deliver packages. The company has set up an indoor obstacle course to test robots from a variety of companies, including Unitree. Eventually, humanoid robots could ride in Amazon’s electric Rivian delivery vans and autonomously deliver packages.

The Humanoid Hub

394,460 views • 1 year ago

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Helix is now learning directly from human video data We have already trained on data collected in the real world, including Brookfield residential units To our knowledge, this is the first instance of a humanoid robot learning navigation end-to-end using only human video

Figure

46,041 views • 11 months ago

Modern AI is confined to the digital world. At Skild AI, we are building towards AGI for the real world, unconstrained by robot type or task — a single, omni-bodied brain. Today, we are sharing our journey, starting with early milestones, with more to come in the weeks ahead. Our Mission: Artificial General Intelligence grounded in the physical world. We believe AGI that can truly understand and reason in the real world can only be built through grounding in the physical world. Our Vision: Any robot, Any task, One brain. We tackle robotics in its full generality – building a continually improving, omni-bodied brain that can control any hardware for any task. Who are we? A passionate group of scientists & engineers driven by our shared vision. We have been researching AI and robotics for more than a decade. Our team includes pioneers of self-supervised learning, curiosity-driven exploration, end-to-end sim2real for visual locomotion, dexterous manipulation, learning from human videos, robot parkour, and many more. Many of these works have won awards at top-tier AI and Robotics conferences. Our team has also built production-ready systems at Anduril, Tesla, Nvidia, Meta, Kitty Hawk, Google, Everyday Robotics, and Amazon. Join us in our mission to build the robot brains of tomorrow.

Modern AI is confined to the digital world. At Skild AI, we are building towards AGI for the real world, unconstrained by robot type or task — a single, omni-bodied brain. Today, we are sharing our journey, starting with early milestones, with more to come in the weeks ahead. Our Mission: Artificial General Intelligence grounded in the physical world. We believe AGI that can truly understand and reason in the real world can only be built through grounding in the physical world. Our Vision: Any robot, Any task, One brain. We tackle robotics in its full generality – building a continually improving, omni-bodied brain that can control any hardware for any task. Who are we? A passionate group of scientists & engineers driven by our shared vision. We have been researching AI and robotics for more than a decade. Our team includes pioneers of self-supervised learning, curiosity-driven exploration, end-to-end sim2real for visual locomotion, dexterous manipulation, learning from human videos, robot parkour, and many more. Many of these works have won awards at top-tier AI and Robotics conferences. Our team has also built production-ready systems at Anduril, Tesla, Nvidia, Meta, Kitty Hawk, Google, Everyday Robotics, and Amazon. Join us in our mission to build the robot brains of tomorrow.

Skild AI

384,982 views • 1 year ago

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

For robots to be useful, they must be able to interact with a wide variety of environments; and yet, scaling interaction data is difficult, expensive, and time consuming. Instead, much research revolves around sim-to-real manipulation — but mostly this has not been mobile manipulation. Recently, though, this has begun to change. Two recent papers from Tairan He and Haoru Xue show us how to unlock the potential of this technique, building policies which, without any real data at all, can move objects around in the world and open doors in the real world with a humanoid robot. Watch Episode #60 of RoboPapers now to learn more, hosted by Chris Paxton and Jiafei Duan. In this episode, we cover two papers:. First is VIRAL: Visual Sim-to-Real at Scale for Humanoid Loco-Manipulation; and second is DoorMan: Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer.

RoboPapers

30,767 views • 7 months ago

Meet Any2Track, a new two-stage RL framework designed to address a core challenge for humanoid robots: reliably tracking diverse, dynamic motions in the real world. The framework consists of two key components: AnyTracker, a general motion tracker, and AnyAdapter, a module that allows the robot to adapt to real-world disturbances like uneven terrain or external forces. This approach gives robots the stability needed for practical use. The system has been successfully deployed on a Unitree G1 humanoid robot with zero-shot sim2real transfer, performing exceptionally well in various motion tracking tasks under multiple real-world disturbances. Project: Codebase:

Meet Any2Track, a new two-stage RL framework designed to address a core challenge for humanoid robots: reliably tracking diverse, dynamic motions in the real world. The framework consists of two key components: AnyTracker, a general motion tracker, and AnyAdapter, a module that allows the robot to adapt to real-world disturbances like uneven terrain or external forces. This approach gives robots the stability needed for practical use. The system has been successfully deployed on a Unitree G1 humanoid robot with zero-shot sim2real transfer, performing exceptionally well in various motion tracking tasks under multiple real-world disturbances. Project: Codebase:

RoboHub🤖

61,378 views • 11 months ago

[1/16] The real world is large, and we want our AI models to operate on the scale of reality. Over the past two years, my colleagues and I have developed fVDB, a deep learning framework for large-scale, high performance spatial intelligence. We finally announced this work at SIGGRAPH and have released the framework in early access. Here’s a small tour of what fVDB is, and what we’ve used it to do.

[1/16] The real world is large, and we want our AI models to operate on the scale of reality. Over the past two years, my colleagues and I have developed fVDB, a deep learning framework for large-scale, high performance spatial intelligence. We finally announced this work at SIGGRAPH and have released the framework in early access. Here’s a small tour of what fVDB is, and what we’ve used it to do.

Francis Williams

34,026 views • 2 years ago

Today we're excited to share a glimpse of what we're building at Generalist. As a first step towards our mission of making general-purpose robots a reality, we're pushing the frontiers of what end-to-end AI models can achieve in the real world. Here's a preview of our early results in autonomous general-purpose dexterous capabilities – fast, reactive, smooth, precise, bi-manual coordinated sensorimotor control.

Today we're excited to share a glimpse of what we're building at Generalist. As a first step towards our mission of making general-purpose robots a reality, we're pushing the frontiers of what end-to-end AI models can achieve in the real world. Here's a preview of our early results in autonomous general-purpose dexterous capabilities – fast, reactive, smooth, precise, bi-manual coordinated sensorimotor control.

Generalist

278,597 views • 1 year ago