Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

1X announces their latest reinforcement learning (RL) controller, which unlocks NEO's full-body mobility for home environments, enabling Redwood AI (1X's in-house AI model) to interact with the physical world more naturally and broadly. The unified controller supports walking in any direction, sitting, standing, kneeling, lying down, getting up, and... climbing stairs using stereo RGB vision - critical for navigating real homes. The controller provides an “action interface” through which teleoperation or Redwood AI can interact in a safe, contact-rich manner with the physical world. Traditional walking controllers often rely on hand-crafted "shaping rewards" to produce human-like gaits, which are time-consuming and don’t scale well across tasks or movement directions. To address this, the team used motion capture data to guide the learning process. The RL system is trained to track kinematic reference trajectories derived from natural human motion while maintaining balance and rhythm in real time. To improve general-purpose utility, the controller goes beyond single-trajectory replay by introducing a two-part design: ⦿ A high-level kinematic planner generates smooth, human-like movement goals from simple input commands (e.g., joystick direction). ⦿ A low-level RL controller tracks these trajectories with dynamic stability. This structure enables smooth transitions between behaviors and resolves the challenge of controlling high-dimensional motion from coarse user input.show more

The Humanoid Hub

63,197 subscribers

113,780 görüntüleme • 1 yıl önce •via X (Twitter)

Sağlık & İyilik Bilim & Teknoloji Eğitim

Anya Rossi• Live Now

Private livecam show

11 Yorum

The Humanoid Hub profil fotoğrafı

The Humanoid Hub1 yıl önce

More context:

Page to Pixel Publishing profil fotoğrafı

Page to Pixel Publishing2 yıl önce

Boost, surf, and weave your way through The Art of Flight, an arcade game about flying multiple ships at the same time. With solo, local co-op, and a leaderboard, there are tons of ways to play. Wishlist on Steam today!

Future mobility profil fotoğrafı

Future mobility1 yıl önce

Cool

MasIp profil fotoğrafı

MasIp1 yıl önce

Slap an A1 logo on there and you have a horror thriller

Arjun Goli 🌎 profil fotoğrafı

Arjun Goli 🌎1 yıl önce

Aesthetics r amazing

cromagnus◀️▶️⏸ profil fotoğrafı

cromagnus◀️▶️⏸1 yıl önce

give this bot a better outfit and it will 10X. their appeal. i get the approach with the drab onesie but they would be getting so much more marketing coverage with more modern custom unique functional fashion sensibilities. its making this amazing bot feel dull.

Brian Bellia profil fotoğrafı

Brian Bellia1 yıl önce

The first 10 seconds of this clip are magical. I can't wait until I can do that with a humanoid. NEO on the sand at the beach ... now, that would really be something.

Don Diego de la Tega profil fotoğrafı

Don Diego de la Tega1 yıl önce

Some of these scenes are CGI, those led on their "ears", are way too bright under daylight to be real.

AdamHumphreys profil fotoğrafı

AdamHumphreys1 yıl önce

I think the mistake made is the lack of padding on the digits of robots. This distributes weight and grip strength leverage. Current models are too rigid but we’ll get there.

vikramvi profil fotoğrafı

vikramvi1 yıl önce

when will you share details like this @Tesla_Optimus $TSLA

Brian Bellia profil fotoğrafı

Brian Bellia1 yıl önce

Given the partnership between 1X and OpenAI, I wonder if there's any role for ChatGPT. I thought it might be able to operate alongside Redwood AI to give NEO a way to verbally interact with humans.

Benzer Videolar

General Intuition CEO Pim de Witte, who's building foundation models trained on video game controller input data ("action-labeled gameplay clips"), says general intelligence won't "taste like an LLM": "We have a scale of data that's going to allow us to jump to the frontier in one capability — which is any system that can be controlled with a game controller (which is most robots) — and then, you can use that to create a sufficiently general intelligence." "As humans, the decision to talk or type is a very, very small subset of the actions that we can actually take." "So in order to create a sufficiently general intelligence to play 10,000+ video games, the model has to be able to predict across the entire action space of human cognition when they're interacting with these environments. Which are 2D and 3D environments, interfaces, long-horizon tasks, short-horizon tasks, [etc.]." "It has to be a sufficiently general intelligence in order to predict actions. Therefore, the type of model you get out is not going to taste like an LLM. This model is going to be incredibly good at navigating unforeseen environments. It's going to be incredibly good at zero-shotting any task that can be done with a game controller."

General Intuition CEO Pim de Witte, who's building foundation models trained on video game controller input data ("action-labeled gameplay clips"), says general intelligence won't "taste like an LLM": "We have a scale of data that's going to allow us to jump to the frontier in one capability — which is any system that can be controlled with a game controller (which is most robots) — and then, you can use that to create a sufficiently general intelligence." "As humans, the decision to talk or type is a very, very small subset of the actions that we can actually take." "So in order to create a sufficiently general intelligence to play 10,000+ video games, the model has to be able to predict across the entire action space of human cognition when they're interacting with these environments. Which are 2D and 3D environments, interfaces, long-horizon tasks, short-horizon tasks, [etc.]." "It has to be a sufficiently general intelligence in order to predict actions. Therefore, the type of model you get out is not going to taste like an LLM. This model is going to be incredibly good at navigating unforeseen environments. It's going to be incredibly good at zero-shotting any task that can be done with a game controller."

TBPN

80,482 görüntüleme • 1 ay önce

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

Jim Fan

124,588 görüntüleme • 2 yıl önce

Physics-based Motion Retargeting from Sparse Inputs paper page: Avatars are important to create interactive and immersive experiences in virtual worlds. One challenge in animating these characters to mimic a user's motion is that commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose. Another challenge is that an avatar might have a different skeleton structure than a human and the mapping between them is unclear. In this work we address both of these challenges. We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies. Our method uses reinforcement learning to train a policy to control characters in a physics simulator. We only require human motion capture data for training, without relying on artist-generated animations for each avatar. This allows us to use large motion capture datasets to train general policies that can track unseen users from real and sparse data in real-time. We demonstrate the feasibility of our approach on three characters with different skeleton structure: a dinosaur, a mouse-like creature and a human. We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available. We discuss and ablate the important components in our framework, specifically the kinematic retargeting step, the imitation, contact and action reward as well as our asymmetric actor-critic observations. We further explore the robustness of our method in a variety of settings including unbalancing, dancing and sports motions.

AK

106,527 görüntüleme • 3 yıl önce

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

The power of generative models — now embodied in humanoids. Announcing DreamControl –– After a year-long research effort at General Robotics — we present a scalable framework for whole-body humanoid control that fuses diffusion priors with reinforcement learning to unlock real-world scene interaction. Diffusion + RL → natural whole-body skills on real robots. DreamControl enables humanoids to move beyond locomotion demos → performing natural, human-like skills such as –– Picking & lifting objects, Opening drawers & doors, Precise punching, kicking, and jumping, Bimanual manipulation tasks Our key innovation: a diffusion prior over human motion that guides RL, eliminating the need for massive teleoperation datasets, and producing motions that look human while transferring to real hardware. Trained purely in simulation, deployed on the Unitree G1 humanoid, DreamControl policies run in real time, bridging sim-to-real with unprecedented naturalness. We leverage a novel hybrid edge + cloud infrastructure that runs RL-trained policies on the edge backed by powerful AI models running in the cloud This is the next step in General Robotics’ journey toward general-purpose humanoid assistants that interact, adapt, and assist autonomously. Paper: Blog: 1/n

Ashish Kapoor

118,133 görüntüleme • 10 ay önce

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

Rohan Paul

37,121 görüntüleme • 5 ay önce

VLAI Robotics Launches High-Dexterity Dual-Arm Robot Starting at ~$5,500 USD VLAI Robotics, utilizing the open-source design of Japan’s OpenArm team, has introduced a high-dexterity, low-cost bimanual robotic system. The product is priced from 39,900 RMB (approximately 5,500 USD), making high-level research tools accessible to institutions and developers. The robot features a human-scale, arm-hand integrated design with up to 16 DOF (8 DOF per arm, including the gripper). It can handle a dual-arm peak payload of 12 kg while maintaining high precision. A key breakthrough is its ability to replicate human upper limb movement trajectories, which enhances data quality for imitation learning and remote operation. VLAI Robotics handled the domestic engineering, manufacturing, and VLA (Vision-Language-Action) algorithm integration, with quality control organized according to the strict standards of the OpenArm R&D framework. The robot is highly adaptable for research, education, and Physical AI training, supporting easy expansion for features like VR teleoperation.

VLAI Robotics Launches High-Dexterity Dual-Arm Robot Starting at ~$5,500 USD VLAI Robotics, utilizing the open-source design of Japan’s OpenArm team, has introduced a high-dexterity, low-cost bimanual robotic system. The product is priced from 39,900 RMB (approximately 5,500 USD), making high-level research tools accessible to institutions and developers. The robot features a human-scale, arm-hand integrated design with up to 16 DOF (8 DOF per arm, including the gripper). It can handle a dual-arm peak payload of 12 kg while maintaining high precision. A key breakthrough is its ability to replicate human upper limb movement trajectories, which enhances data quality for imitation learning and remote operation. VLAI Robotics handled the domestic engineering, manufacturing, and VLA (Vision-Language-Action) algorithm integration, with quality control organized according to the strict standards of the OpenArm R&D framework. The robot is highly adaptable for research, education, and Physical AI training, supporting easy expansion for features like VR teleoperation.

RoboHub🤖

12,961 görüntüleme • 9 ay önce

Agile Continuous Jumping in Discontinuous Terrains discuss: We focus on agile, continuous, and terrain-adaptive jumping of quadrupedal robots in discontinuous terrains such as stairs and stepping stones. Unlike single-step jumping, continuous jumping requires accurately executing highly dynamic motions over long horizons, which is challenging for existing approaches. To accomplish this task, we design a hierarchical learning and control framework, which consists of a learned heightmap predictor for robust terrain perception, a reinforcement-learning-based centroidal-level motion policy for versatile and terrain-adaptive planning, and a low-level model-based leg controller for accurate motion tracking. In addition, we minimize the sim-to-real gap by accurately modeling the hardware characteristics. Our framework enables a Unitree Go1 robot to perform agile and continuous jumps on human-sized stairs and sparse stepping stones, for the first time to the best of our knowledge. In particular, the robot can cross two stair steps in each jump and completes a 3.5m long, 2.8m high, 14-step staircase in 4.5 seconds. Moreover, the same policy outperforms baselines in various other parkour tasks, such as jumping over single horizontal or vertical discontinuities.

Agile Continuous Jumping in Discontinuous Terrains discuss: We focus on agile, continuous, and terrain-adaptive jumping of quadrupedal robots in discontinuous terrains such as stairs and stepping stones. Unlike single-step jumping, continuous jumping requires accurately executing highly dynamic motions over long horizons, which is challenging for existing approaches. To accomplish this task, we design a hierarchical learning and control framework, which consists of a learned heightmap predictor for robust terrain perception, a reinforcement-learning-based centroidal-level motion policy for versatile and terrain-adaptive planning, and a low-level model-based leg controller for accurate motion tracking. In addition, we minimize the sim-to-real gap by accurately modeling the hardware characteristics. Our framework enables a Unitree Go1 robot to perform agile and continuous jumps on human-sized stairs and sparse stepping stones, for the first time to the best of our knowledge. In particular, the robot can cross two stair steps in each jump and completes a 3.5m long, 2.8m high, 14-step staircase in 4.5 seconds. Moreover, the same policy outperforms baselines in various other parkour tasks, such as jumping over single horizontal or vertical discontinuities.

AK

35,794 görüntüleme • 1 yıl önce

The World Model as NEO's Cognitive Core 1X has revealed a major AI development where the NEO humanoid can translate any natural language prompt into robotic action. It demonstrates this capability even for novel tasks, objects, and environments not found in its robot dataset. - the 1X World Model is trained on internet-scale human interaction videos and fine-tuned with robot data to ground its understanding in physics and in NEO's embodiment - from a simple voice or text prompt, the world model generates a visualization of future actions - a built-in inverse dynamics model then translates these into precise motor movements for NEO

The World Model as NEO's Cognitive Core 1X has revealed a major AI development where the NEO humanoid can translate any natural language prompt into robotic action. It demonstrates this capability even for novel tasks, objects, and environments not found in its robot dataset. - the 1X World Model is trained on internet-scale human interaction videos and fine-tuned with robot data to ground its understanding in physics and in NEO's embodiment - from a simple voice or text prompt, the world model generates a visualization of future actions - a built-in inverse dynamics model then translates these into precise motor movements for NEO

The Humanoid Hub

68,453 görüntüleme • 7 ay önce

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

In order for robots to be deployed in the real world, performing tasks of real value, they must be reliable. Unfortunately, even more, most robotic demos work maybe 70-80% of the time at best. The way to get better reliability is to do real-world reinforcement learning: having the robot teach itself how to perform the task up to a high level of success. The key to doing this is to start with a core of expert human data, use that to train a policy then iteratively improve it, until finally finishing with on-policy reinforcement learning. Kun Lei talks through a unified framework for imitation and reinforcement learning based on PPO, which enables this improvement process. In this episode, Kun Lei explains the theory behind his reinforcement learning method and how it allowed his robot to run in a shopping mall juicing oranges for seven hours at a time, among experiments on a wide variety of tasks and embodiments. Watch episode 58 of RoboPapers now, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

18,813 görüntüleme • 7 ay önce

🤔 Ever wondered if simulation-based animation/avatar learnings can be applied to real humanoid in real-time? 🤖 Introducing H2O (Human2HumanOid): - 🧠 An RL-based human-to-humanoid real-time whole-body teleoperation framework - 💃 Scalable retargeting and training using large human motion dataset - 🎥 With just an RGB camera, everyone can teleoperate a full-sized humanoid to perform actions like pick and place, walking, kicking, boxing, etc - 💡Unleash the potential of humanoids with human cognitive skills and adaptability 🔗: 📄: 🎬: H2O proposes: - A scalable retargeting framework for obtaining large-scale humanoid motion dataset, intelligently filtering out infeasible motion for the humanoid embodiment. - We train a full-body motion imitator (similar to PHC) and deploy to the real world zero-shot. - Using this framework, we enable real-time teleoperation of a humanoid via a human operator and webcam, performing skills such as pick and place, kicking, walking strollers, etc. Team: Tairan He Zhengyi “Zen” Luo Wenli Xiao @ChongZitaZhang Kris Kitani Changliu Liu Guanya Shi

Zhengyi “Zen” Luo

47,323 görüntüleme • 2 yıl önce

Today, we're announcing the next evolution of Sword Thrive: a fully revamped experience anchored by a new conversational care model, Vision AI, and a more powerful Phoenix. At the center of this evolution is a first-of-its-kind care model: a 3-way chat connecting the member, Phoenix, and their Doctor of Physical Therapy in a single interface, making continuous, 24/7 care possible for the first time, with the clinical oversight only a human specialist can provide. Vision AI, our proprietary technology trained on years of movement data, now delivers precise movement detection in real-world home environments - cluttered, noisy, and everything in between - and runs entirely through the Sword Health app on smartphones and tablets. Phoenix, our AI Care Specialist, is now more capable as well, providing deeper, personalized two-way conversational guidance and real-time form correction, adapting to each member as they move. From proactive care to continuous recovery, this is how healthcare should show up: with reliability, clinical memory, and real-time support that never drops the thread. Watch the video to see it in action:

Today, we're announcing the next evolution of Sword Thrive: a fully revamped experience anchored by a new conversational care model, Vision AI, and a more powerful Phoenix. At the center of this evolution is a first-of-its-kind care model: a 3-way chat connecting the member, Phoenix, and their Doctor of Physical Therapy in a single interface, making continuous, 24/7 care possible for the first time, with the clinical oversight only a human specialist can provide. Vision AI, our proprietary technology trained on years of movement data, now delivers precise movement detection in real-world home environments - cluttered, noisy, and everything in between - and runs entirely through the Sword Health app on smartphones and tablets. Phoenix, our AI Care Specialist, is now more capable as well, providing deeper, personalized two-way conversational guidance and real-time form correction, adapting to each member as they move. From proactive care to continuous recovery, this is how healthcare should show up: with reliability, clinical memory, and real-time support that never drops the thread. Watch the video to see it in action:

Sword Health

27,974 görüntüleme • 5 ay önce

We believe we’re the first robotics company to demonstrate a robot peeling an apple with dual dexterous human-like hands. This breakthrough closes a key gap in robotics, achieving bimanual, contact-rich manipulation and moving far beyond the limits of simple grippers. 🧵↓ Today’s AI models (VLMs) are excellent at perception but struggle with action. Controlling high-degree-of-freedom hands for tasks like this is incredibly complex, and precise finger-level teleoperation is nearly impossible for humans. Our first step was a shared-autonomy system: rather than controlling every finger, the operator triggers pre-learned skills like a “rotate apple or tennis ball” primitive via a keyboard press or pedal. This makes scalable data collection and RL training possible. How does the AI manage this? We created "MoDE-VLA" (Mixture of Dexterous Experts). It fuses vision, language, force, and touch data by using a team of specialist "experts," making control in high-dimensional spaces stable and effective. The combination of these two innovations allows for seamless, contact-rich manipulation. The human provides high-level guidance, and the robot executes the complex in-hand coordination required. This work paves the way for robots that can safely handle delicate tasks in human environments. Want the full technical details? 📄 Read the full research paper: Visit us at NVIDIA GTC Booth #1838, Hall 3 to learn more! #Robotics #AI #DexterousManipulation #VLA #NVIDIAGTC Nancy Villicaña NVIDIA GTC

We believe we’re the first robotics company to demonstrate a robot peeling an apple with dual dexterous human-like hands. This breakthrough closes a key gap in robotics, achieving bimanual, contact-rich manipulation and moving far beyond the limits of simple grippers. 🧵↓ Today’s AI models (VLMs) are excellent at perception but struggle with action. Controlling high-degree-of-freedom hands for tasks like this is incredibly complex, and precise finger-level teleoperation is nearly impossible for humans. Our first step was a shared-autonomy system: rather than controlling every finger, the operator triggers pre-learned skills like a “rotate apple or tennis ball” primitive via a keyboard press or pedal. This makes scalable data collection and RL training possible. How does the AI manage this? We created "MoDE-VLA" (Mixture of Dexterous Experts). It fuses vision, language, force, and touch data by using a team of specialist "experts," making control in high-dimensional spaces stable and effective. The combination of these two innovations allows for seamless, contact-rich manipulation. The human provides high-level guidance, and the robot executes the complex in-hand coordination required. This work paves the way for robots that can safely handle delicate tasks in human environments. Want the full technical details? 📄 Read the full research paper: Visit us at NVIDIA GTC Booth #1838, Hall 3 to learn more! #Robotics #AI #DexterousManipulation #VLA #NVIDIAGTC Nancy Villicaña NVIDIA GTC

Sharpa

20,429 görüntüleme • 5 ay önce

Brain-controlled exoskeletons to train humanoid robots! 🧠 Fourier just presented human tele-operators using brain control interfaces and exoskeletal arms to train humanoid robots on home tasks. The brain control interface is the interesting part. Instead of using a controller or joystick to teleoperate, the operator's movements and intentions are captured more naturally through the exoskeleton and BCI. This means the demonstrations are more fluid, more human-like, and better suited for training robots to perform delicate home tasks. Multiple tele-operators are simultaneously generating training data across multiple robots. This is how you build the dataset needed for eventual full autonomy, without waiting years for it to arrive. This might be the bridge between "robots that work in controlled environments" and "robots that work in homes." Not full autonomy right away, but trusted human intelligence operating through a robot body, getting better with every task completed. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Brain-controlled exoskeletons to train humanoid robots! 🧠 Fourier just presented human tele-operators using brain control interfaces and exoskeletal arms to train humanoid robots on home tasks. The brain control interface is the interesting part. Instead of using a controller or joystick to teleoperate, the operator's movements and intentions are captured more naturally through the exoskeleton and BCI. This means the demonstrations are more fluid, more human-like, and better suited for training robots to perform delicate home tasks. Multiple tele-operators are simultaneously generating training data across multiple robots. This is how you build the dataset needed for eventual full autonomy, without waiting years for it to arrive. This might be the bridge between "robots that work in controlled environments" and "robots that work in homes." Not full autonomy right away, but trusted human intelligence operating through a robot body, getting better with every task completed. ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

20,898 görüntüleme • 6 ay önce

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

🚨 BREAKING: Microsoft's first robotics foundation model! 🤯 Microsoft just announced Rho-alpha (ρα), their first robotics model derived from the Phi series of vision-language models. Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks. Commands like "push the green button with the right gripper," "pull out the red wire," "flip the top switch on," or "turn the knob to position 5" get executed directly by dual-arm robots. What makes this different from standard vision-language-action (VLA) models is the additional modalities. Rho-alpha is a VLA+ model that adds tactile sensing to the perceptual mix, with plans to incorporate force feedback. On the learning side, the model is designed to continually improve during deployment by learning from human feedback. The training approach combines trajectories from physical demonstrations and simulated tasks with web-scale visual question answering data. Since teleoperation data is scarce and expensive, Microsoft is using NVIDIA Isaac Sim on Azure to generate physically accurate synthetic datasets via reinforcement learning. These simulated trajectories get combined with commercial and open physical demonstration datasets. The model is currently under evaluation on dual-arm setups and humanoid robots. Microsoft is opening an Early Access Program for organizations interested in evaluating Rho-alpha. Robots that can adapt to dynamic situations and human preferences are more useful in real environments and more trusted by the people operating them. Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

60,928 görüntüleme • 6 ay önce

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

How do you give a humanoid the general motion capability? Not just single motions, but all motion? Introducing SONIC, our new work on supersizing motion tracking for natural humanoid control. We argue that motion tracking is the scalable foundation task for humanoids. So we "supersized" it: 9k+ GPU hours and 100M+ motion frames. But tracking alone is not enough; we show how to make a useful control system out of it: - Universal Kinematic Planner: Enables game-like gamepad control and high-level teleoperation, just like controlling a character in a game. - VR Full-Body Teleop: Direct, real-time whole-body control by a human wearing a VR headset. - VR Keypoint Teleop: Control the upper body (hands/head) while our planner handles robust locomotion automatically. - VLA Integration: We connect this motion tracker to autonomous Visual-Language-Action (VLA) models for autonomous task execution! We use a Universal Token Space to UNIFY this command space, turning our robust tracker into a general-purpose, programmable humanoid brain. This is the generalist "System 1" for humanoids. 🚀 Project: #Humanoids #Robotics #AI #FoundationModels #NVIDIAResearch 🧠🔥

Zhengyi “Zen” Luo

62,208 görüntüleme • 9 ay önce

Let's reverse engineer Disney's adorable, lifelike robot! I couldn't find a whitepaper, but this is how I think it's trained: 1. The emotional behaviors are curated by Disney animation artists, keyframe by keyframe. But it cannot be "rendered" directly on the robot because it doesn't take into account the complex real-world physics. 2. Reinforcement learning (RL) is a great tool for training low-level robot controllers. RL needs a reward function to optimize, and it's typically a task reward (e.g. walk in a straight line as fast as possible). The problem is that RL doesn't know what counts as "natural behavior", and often produces weird-looking body postures that somehow still maximize the reward. This is a human alignment problem just like ChatGPT. 3. Enters Adversarial Motion Prior (AMP): a technique that learns the human preference by training a classifier on what we consider "emotional & cute". In GAN literature, this is called a discriminator. Disney artists are good at creating such a dataset. You can then add AMP as an auxiliary reward in simulation to nudge the robot towards desired behaviors. AMP was developed by Peng et al. 2021 and Escontrela et al. 2022. 4. Add lots of data augmentation to make the controller robust to physical disturbances. In RL, it's called "domain randomization". This is a very powerful technique that bridges the gap between simulator and reality. Previously, OpenAI used domain randomization to train a 5-finger robot hand to manipulate a Rubik's Cube: IEEE news article gave hints about the pipeline: Finally, praying for world peace 🙏. I hope robotics like this will bring more joy to the world.

Let's reverse engineer Disney's adorable, lifelike robot! I couldn't find a whitepaper, but this is how I think it's trained: 1. The emotional behaviors are curated by Disney animation artists, keyframe by keyframe. But it cannot be "rendered" directly on the robot because it doesn't take into account the complex real-world physics. 2. Reinforcement learning (RL) is a great tool for training low-level robot controllers. RL needs a reward function to optimize, and it's typically a task reward (e.g. walk in a straight line as fast as possible). The problem is that RL doesn't know what counts as "natural behavior", and often produces weird-looking body postures that somehow still maximize the reward. This is a human alignment problem just like ChatGPT. 3. Enters Adversarial Motion Prior (AMP): a technique that learns the human preference by training a classifier on what we consider "emotional & cute". In GAN literature, this is called a discriminator. Disney artists are good at creating such a dataset. You can then add AMP as an auxiliary reward in simulation to nudge the robot towards desired behaviors. AMP was developed by Peng et al. 2021 and Escontrela et al. 2022. 4. Add lots of data augmentation to make the controller robust to physical disturbances. In RL, it's called "domain randomization". This is a very powerful technique that bridges the gap between simulator and reality. Previously, OpenAI used domain randomization to train a 5-finger robot hand to manipulate a Rubik's Cube: IEEE news article gave hints about the pipeline: Finally, praying for world peace 🙏. I hope robotics like this will bring more joy to the world.

Jim Fan

314,694 görüntüleme • 2 yıl önce

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

NEWS: Humanoid robotics company Figure has released Helix 02, what they claim in their most capable humanoid model yet. "A single neural system that controls the full body directly from pixels, enabling dexterous, long horizon autonomy across an entire room: • Autonomous, long‑horizon loco-manipulation: Helix 02 unloads and reloads a dishwasher across a full-sized kitchen - a four-minute, end-to-end autonomous task that integrates walking, manipulation, and balance with no resets and no human intervention. We believe this is the longest horizon, most complex task completed autonomously by a humanoid robot to date. • All sensors in. All actuators out: Helix 02 connects every onboard sensor - vision, touch, and proprioception - directly to every actuator through a single unified visuomotor neural network. • Human-like whole body control from human data: All results are enabled by System 0, a learned whole‑body controller trained on over 1,000 hours of human motion data and sim‑to‑real reinforcement learning. System 0 replaces 109,504 lines of hand‑engineered C++ with a single neural prior for stable, natural motion. • New classes of dexterity: With Figure 03’s embedded tactile sensing and palm cameras, Helix 02 performs manipulation that was previously out of reach: extracting individual pills, dispensing precise syringe volumes, and singulating small, irregular objects from clutter despite self‑occlusion. Helix 02 is trained on over 1,000 hours of human motion data and integrates vision, touch, and proprioception."

Sawyer Merritt

624,770 görüntüleme • 6 ay önce

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the motion of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,514 görüntüleme • 2 yıl önce

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 görüntüleme • 6 ay önce

NEW RESEARCH: You can now create a new robot optimized for any given task! I love this new project by Huy Ha, Shuran Song, and others. Called "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design", it generates a robot's physical design and its controller together from a task spec. DEFINITIONS: - Reward function: A scoring rule that assigns a number to how well a behavior achieves the task. Here, it is the objective the generated design is pushed to maximize (e.g., track the target motion with low error). - Tokenizing: dividing continuous or structured data (a robot's links, joints, motor specs, states, actions) into a discrete vocabulary of symbols a transformer can process, the same step that turned pixels and audio into "language" for these models. - Diffusion transformer (DiT): A transformer trained to turn random noise into structured output through iterative denoising. Here, it generates robot bodies and trajectories instead of images. - MuJoCo: The standard fast physics simulator for robotics research (DeepMind-maintained). The Menagerie is its curated zoo of ready-to-use robot models. - CMA-ES: Covariance Matrix Adaptation Evolution Strategy, the workhorse black-box optimizer: it evolves a population of candidate designs, keeps the best, and needs thousands of simulator rollouts. - Bimanual multi-trajectory optimization: Finding one design/controller that performs well across several target motions for a two-armed robot at once, harder than optimizing for a single arm and a single motion. - BERT/MAE masked-modeling trick: Train one model to fill in whatever parts of the input you hide (words for BERT, image patches for MAE); at inference, choosing what to mask chooses the task, so masking the body makes it a designer and masking the actions makes it a controller. In practice, you give it a target end-effector motion and a reward function, and it outputs a complete embodiment (link, joint, motor, and inertial property), as well as a controller to drive it. It works by tokenizing both the body (links/joints/motors) and the dynamics (states/actions) into a compact scheme called RoboTokens, training a diffusion transformer (DiT) over them. The same model predicts dynamics using those predictions ("Dynamics Self-Guidance") to push generated designs toward higher reward at inference time. Masking different token types (using the BERT/MAE masked-modeling trick) lets the one model do three jobs: generate an embodiment, control an arbitrary embodiment, or design one conditioned on a motion. It is trained on 11 robots from the MuJoCo Menagerie (0.65 kg hand to 67.5 kg quadruped, 6–35 joints), and validated in sim and on a physical ALOHA doing cloth flinging. I like the fact that this approach inverts the entire recent robotics ideas: designing a policy for a fixed robot -> designing the robot for a fixed task. Every other approach assumes the body is given and learns a controller. Transformer Transformer takes the task (target motion + reward), then generates the body and controller jointly. In practice, it is a ~180× speedup over the standard optimizer at equal-or-better quality. It reaches "CMA-ES-level quality in seconds" and finishes bimanual multi-trajectory optimization in that is worth underlining nowadays! Also worth mentioning: this is the lab behind UMI and Handroid, that I mentioned here previously! The team seems extremely creative, i love these out-of-the-box approaches. Enjoy watching the demo of robot optimization in 3D, data acquisition, then real-life testing:

NEW RESEARCH: You can now create a new robot optimized for any given task! I love this new project by Huy Ha, Shuran Song, and others. Called "Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design", it generates a robot's physical design and its controller together from a task spec. DEFINITIONS: - Reward function: A scoring rule that assigns a number to how well a behavior achieves the task. Here, it is the objective the generated design is pushed to maximize (e.g., track the target motion with low error). - Tokenizing: dividing continuous or structured data (a robot's links, joints, motor specs, states, actions) into a discrete vocabulary of symbols a transformer can process, the same step that turned pixels and audio into "language" for these models. - Diffusion transformer (DiT): A transformer trained to turn random noise into structured output through iterative denoising. Here, it generates robot bodies and trajectories instead of images. - MuJoCo: The standard fast physics simulator for robotics research (DeepMind-maintained). The Menagerie is its curated zoo of ready-to-use robot models. - CMA-ES: Covariance Matrix Adaptation Evolution Strategy, the workhorse black-box optimizer: it evolves a population of candidate designs, keeps the best, and needs thousands of simulator rollouts. - Bimanual multi-trajectory optimization: Finding one design/controller that performs well across several target motions for a two-armed robot at once, harder than optimizing for a single arm and a single motion. - BERT/MAE masked-modeling trick: Train one model to fill in whatever parts of the input you hide (words for BERT, image patches for MAE); at inference, choosing what to mask chooses the task, so masking the body makes it a designer and masking the actions makes it a controller. In practice, you give it a target end-effector motion and a reward function, and it outputs a complete embodiment (link, joint, motor, and inertial property), as well as a controller to drive it. It works by tokenizing both the body (links/joints/motors) and the dynamics (states/actions) into a compact scheme called RoboTokens, training a diffusion transformer (DiT) over them. The same model predicts dynamics using those predictions ("Dynamics Self-Guidance") to push generated designs toward higher reward at inference time. Masking different token types (using the BERT/MAE masked-modeling trick) lets the one model do three jobs: generate an embodiment, control an arbitrary embodiment, or design one conditioned on a motion. It is trained on 11 robots from the MuJoCo Menagerie (0.65 kg hand to 67.5 kg quadruped, 6–35 joints), and validated in sim and on a physical ALOHA doing cloth flinging. I like the fact that this approach inverts the entire recent robotics ideas: designing a policy for a fixed robot -> designing the robot for a fixed task. Every other approach assumes the body is given and learns a controller. Transformer Transformer takes the task (target motion + reward), then generates the body and controller jointly. In practice, it is a ~180× speedup over the standard optimizer at equal-or-better quality. It reaches "CMA-ES-level quality in seconds" and finishes bimanual multi-trajectory optimization in that is worth underlining nowadays! Also worth mentioning: this is the lab behind UMI and Handroid, that I mentioned here previously! The team seems extremely creative, i love these out-of-the-box approaches. Enjoy watching the demo of robot optimization in 3D, data acquisition, then real-life testing:

Léo

25,735 görüntüleme • 8 gün önce