Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Command humanoids directly with natural language? Introducing LangWBC, a generative, end-to-end policy that turns natural language into real-world whole-body humanoid control! 💬→🦿Smooth, robust, surprisingly intuitive! See more 👉 #RSS2025

Zhongyu Li

1,497 subscribers

14,142 views • 1 year ago •via X (Twitter)

Science & Technology Education #RSS2025

Anya Rossi• Live Now

Private livecam show

9 Comments

Zhongyu Li1 year ago

Intuition of LangWBC: Map language to whole-body motion, using a CVAE for its structured latent space→Enables generalization to linguistic variations & unseen latents! 1. Train lang.-agnostic RL experts for motion tracking 2. Distill into a single e2e policy conditioned on lang.

Zhongyu Li1 year ago

Real-world deployment shows LangWBC in action: ✅ Agile, versatile whole-body motion ✅ Smooth transitions—e.g., walking ➡️ running ✅ Emergent motions via latent space blending ("walk" + "sidestep" = diagonal!) ✅ Easy integration with LLMs for natural interaction

Zhongyu Li1 year ago

📄 This work has been accepted to #RSS2025! Check out the full paper here: Big thanks to our collaborators @Shaoyy2003 @x_h_ucb @qiayuanliao @t_k_233 @KoushilSreenath and everyone who supported this work!

TXYZ1 year ago

⏰ Countdown alert! 🚀 Get free access to our new AI Writing feature before Dec 1st! Don't miss your chance to explore its power. Try it now—no cost, no catch! 🔗

LECCA Intern (Ø,G) 🍜1 year ago

Language-to-motion in one shot. LangWBC feels like GPT for humanoids.

Sara Orbgirl1 year ago

Imagine commanding robots with words, like a sci-fi dream come true! 🌠 Who knew talking could create motion? Impressive work! 🤖

Shi Soul1 year ago

seems the feasible motions are too few( walk, run, wave) can we do sth more?

Weishuai Zeng1 year ago

Nice work Zhongyu! Congratulations~ May I know how do you obtain base linear velocity in real world as you mention in your paper? QwQ

Sanjar1 year ago

Great work! No codebase yet?

Related Videos

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

TextOp: A real-time text-to-motion framework for humanoid robots. It allows users to instruct the robot using natural language and to modify commands on the fly, producing smooth, whole-body motions instantly.

The Humanoid Hub

25,125 views • 7 months ago

Meet Meter Command. A generative UI product for getting information, taking action, and building real-time software – all in natural language.

Meet Meter Command. A generative UI product for getting information, taking action, and building real-time software – all in natural language.

Meter

332,187 views • 1 year ago

Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥

Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥

Zhengyi “Zen” Luo

15,851 views • 7 months ago

Two weeks ago, Figure's CEO announced the end of their partnership with OpenAI and promised to show the in-house AI development – here we are: Helix is a model that unifies perception, language and learned control to execute humanoid tasks by following natural language prompts.

Two weeks ago, Figure's CEO announced the end of their partnership with OpenAI and promised to show the in-house AI development – here we are: Helix is a model that unifies perception, language and learned control to execute humanoid tasks by following natural language prompts.

The Humanoid Hub

69,415 views • 1 year ago

Introducing Figure 02, a humanoid robot capable of natural language conversations thanks to OpenAI. What do you think?

Introducing Figure 02, a humanoid robot capable of natural language conversations thanks to OpenAI. What do you think?

MIT CSAIL

67,522 views • 1 year ago

New paper introduces NaVILA, a vision-language-action (VLA) model that integrates high-level visual-language understanding and low-level locomotion control. It enables humanoid or quadruped robots to navigate unseen environments with natural language instructions.

New paper introduces NaVILA, a vision-language-action (VLA) model that integrates high-level visual-language understanding and low-level locomotion control. It enables humanoid or quadruped robots to navigate unseen environments with natural language instructions.

The Humanoid Hub

20,430 views • 1 year ago

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

Zhengyi “Zen” Luo

61,889 views • 7 months ago

Mercury Command is live. AI built directly into your Mercury account that completes financial work. Natural language, real actions, and you approve everything. This is how banking* works now.

Mercury Command is live. AI built directly into your Mercury account that completes financial work. Natural language, real actions, and you approve everything. This is how banking* works now.

Mercury

142,436 views • 7 days ago

LMDrive Closed-Loop End-to-End Driving with LLM An end-to-end, closed-loop, language-based autonomous driving framework, which interacts with the dynamic environment via multi-modal multi-view sensor data and natural language instructions

LMDrive Closed-Loop End-to-End Driving with LLM An end-to-end, closed-loop, language-based autonomous driving framework, which interacts with the dynamic environment via multi-modal multi-view sensor data and natural language instructions

rsasaki0109

10,085 views • 2 years ago

LimX Dynamics unveiled COSA, a unified brain and body OS for humanoid robots. It lets their OLI robot think and act in real time from walking to manipulating objects using natural language. Cognition and control, fused.

LimX Dynamics unveiled COSA, a unified brain and body OS for humanoid robots. It lets their OLI robot think and act in real time from walking to manipulating objects using natural language. Cognition and control, fused.

XMAQUINA

15,054 views • 5 months ago

Big news from the TeleAI embodied intelligence team! They unveiled TextOp, a general cerebellar framework for real-time, text-driven humanoid control. TextOp lets humans control the robot via natural language, dynamically modifying instructions during runtime to instantly generate smooth, whole-body actions. This enables precise and highly versatile control. The framework uses a two-layer architecture: an action diffusion model (High-Level) and a general motion tracking policy (Low-Level). It’s a new paradigm that eliminates the need for pre-recorded scripts or manual programming.

Big news from the TeleAI embodied intelligence team! They unveiled TextOp, a general cerebellar framework for real-time, text-driven humanoid control. TextOp lets humans control the robot via natural language, dynamically modifying instructions during runtime to instantly generate smooth, whole-body actions. This enables precise and highly versatile control. The framework uses a two-layer architecture: an action diffusion model (High-Level) and a general motion tracking policy (Low-Level). It’s a new paradigm that eliminates the need for pre-recorded scripts or manual programming.

RoboHub🤖

23,804 views • 7 months ago

🚨 Your test cases shouldn’t break every time your UI changes. KaneAI writes, runs, and self-heals end-to-end tests directly from natural language built for real QA teams, not toy demos. From JIRA tickets to production-ready test code.

🚨 Your test cases shouldn’t break every time your UI changes. KaneAI writes, runs, and self-heals end-to-end tests directly from natural language built for real QA teams, not toy demos. From JIRA tickets to production-ready test code.

Darshal Jaitwar

70,504 views • 4 months ago

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6 #Robotics #AI #ZeroShot #Manipulation

🤖What if a robot could perform a new task just from a natural language command, with zero demonstrations? Our new work, NovaFlow, makes it possible! We use pre-trained video generative model to create a video of the task, then translate it into a plan for real-world robot execution. 1/6 #Robotics #AI #ZeroShot #Manipulation

Hongyu Li

105,442 views • 8 months ago

Today we’re releasing GitHub Spark — a new tool in Copilot that turns your ideas into full-stack apps, entirely in natural language.

Today we’re releasing GitHub Spark — a new tool in Copilot that turns your ideas into full-stack apps, entirely in natural language.

Satya Nadella

3,172,222 views • 11 months ago

Introducing Comet Shortcuts You can now automate repetitive boring web workflows with simple natural language prompts and access them with a /command anywhere We envision a future where people will be able to share and monetize custom shortcuts they built. More on that soon.

Introducing Comet Shortcuts You can now automate repetitive boring web workflows with simple natural language prompts and access them with a /command anywhere We envision a future where people will be able to share and monetize custom shortcuts they built. More on that soon.

Aravind Srinivas

164,515 views • 10 months ago

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

NVIDIA researchers present SONIC, a generalist humanoid controller: It scales motion tracking on a single policy to achieve natural, robust whole-body movement. The scalable foundation avoids manual reward engineering and features a universal token space and kinematic planner to seamlessly integrate diverse inputs, including VR/video teleoperation and VLA foundation models, for real-world tasks.

The Humanoid Hub

31,445 views • 7 months ago

Reading research papers shouldn't feel like learning a new language Introducing real-time AI voice chat for arXiv papers 🎙️ Talk directly about any research paper in any language. Ask questions and request explanations, all through natural conversation 🚀

Reading research papers shouldn't feel like learning a new language Introducing real-time AI voice chat for arXiv papers 🎙️ Talk directly about any research paper in any language. Ask questions and request explanations, all through natural conversation 🚀

alphaXiv

10,473 views • 1 year ago

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Introducing Omnigrasp: Grasping Diverse Objects with Simulated Humanoids. With Omnigrasp, we show that we can control a humanoid equipped with dexterous hands to grasp diverse objects (>1200) and follow diverse trajectories, with one policy! 🌐: 📜:

Zhengyi “Zen” Luo

92,104 views • 1 year ago

‘CLONE’ – whole-body teleoperation of a humanoid. Intuitive control signals are captured by tracking the teleoperator’s head and hand poses using Apple Vision Pro. A Mixture-of-Experts policy takes the sparse input and synthesizes the corresponding whole-body humanoid pose.

‘CLONE’ – whole-body teleoperation of a humanoid. Intuitive control signals are captured by tracking the teleoperator’s head and hand poses using Apple Vision Pro. A Mixture-of-Experts policy takes the sparse input and synthesizes the corresponding whole-body humanoid pose.

The Humanoid Hub

38,155 views • 1 year ago