Zhengyi “Zen” Luo
@zhengyiluo • 5,908 subscribers
Research Scientist, GEAR @NvidiaAI | PhD @CMU_Robotics | Founder @CirkitDesign | CS @penn Opinions are my own.
Shorts
Videos
SONIC has been released for a while — here are some tips for whole-body teleoperation 🤖 1. WiFi quality directly determines teleoperation performance It's highly network-dependent. Use a dedicated router, avoid shared networks. PICO and the robot/host must be on the same network. High latency = dropped packets = unstable robot!! 2. Tracker orientation and the clothes you wear matters Ankle trackers must have their indicator lights facing up, unobstructed by clothing. Blocked sensors = unsteady footsteps. 3. Watch the posture before resuming teleop After pausing, unpausing immediately executes your current detected body pose. Return to a neutral standing position before resuming. Slight delay is normal — but too much and performance degrades fast!
Zhengyi “Zen” Luo40,079 views • 2 months ago
How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥
Zhengyi “Zen” Luo61,761 views • 6 months ago
PHC has been accepted by ICCV 2023! We aim to develop a physics-based humanoid controller capable of imitating ALL of the motion from the AMASS dataset (almost there🧐), recover from failure state, does NOT use any external forces, all the while supporting real-time use cases!
Zhengyi “Zen” Luo57,938 views • 2 years ago
🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi
Zhengyi “Zen” Luo47,251 views • 2 years ago
Humanoids need a single, generalist control policy for all of their physical tasks, not a new one for every new chore or demo. A policy for walking can't dance. A policy for dancing can't support mowing the lawn. We need to scale up humanoid control for diverse behaviors, just like we did with language and video models. Watch this humanoid, supported by a single control policy, to start its day, mow the lawn, jog, dance, and even hold the elevator for a colleague. This is the "System 1" for humanoids—a robust, natural humanoid controller that will support general-purpose autonomy. Project Page: #Robotics #AI #NVIDIA #SONIC #FoundationModel 🤖🔥
Zhengyi “Zen” Luo15,843 views • 6 months ago
No more content to load