Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

See Spot perform dynamic whole-body manipulation. Using a combination of reinforcement learning (RL) and sampling-based control, the robot is able to autonomously drag, roll, and stack tires weighing 15 kg (33 lb), well above its maximum arm lift capacity. Learn more about coordinating locomotion and manipulation processes:

RAI Institute

7,885 subscribers

87,700 Aufrufe • vor 11 Monaten •via X (Twitter)

Wissenschaft & Technologie Bildung Sport

Anya Rossi• Live Now

Private livecam show

17 Kommentare

Profilbild von Flesh & Bone VR

Flesh & Bone VRvor 11 Monaten

That leg up was super impressive. You can't teach these non humanoids with mocap.

Profilbild von Олег Сельдюгаев

Олег Сельдюгаевvor 11 Monaten

This video makes it clear that a second hand is necessary for quick and precise loading and unloading operations. The second hand can be simpler than the primary hand.

Profilbild von iRiSh

iRiShvor 11 Monaten

I hope I live long enough to see the future where these are everywhere. I want one. Everyone wants one

Profilbild von Kwasikot

Kwasikotvor 11 Monaten

The most amazing dog! What an incredible achievement! 😊

Profilbild von BadassRockets

BadassRocketsvor 11 Monaten

Two arms with guns Charging kennels Secured borders

Profilbild von Jo

Jovor 11 Monaten

Sandbag houses is the best use case. Whole houses built one bag at a time with a Spot bot.

Profilbild von Dennis

Dennisvor 11 Monaten

Quadruped with one arm trying to train in novel ways to manipulate things that we already have plenty of human based training data, is rather inneficient.

Profilbild von Sammy

Sammyvor 11 Monaten

It is doing it exceptionally well.

Profilbild von Max's Ghost

Max's Ghostvor 11 Monaten

That's great. Come back when he can mount and balance it.

Profilbild von furtive pygmy (parody) 💨

furtive pygmy (parody) 💨vor 11 Monaten

tires with ir markers on them are so futuristic, also narrow sidewalls

Profilbild von Mick Russom

Mick Russomvor 11 Monaten

yeah, now tell people about the abysmal battery life.

Profilbild von Marie

Marievor 11 Monaten

Say NO to Robots! 🤖 Join the ARA.. fight for humanity! 😉

Profilbild von Anto Patrex

Anto Patrexvor 11 Monaten

This is pretty impressive, specially with one arm.

Profilbild von All4000ft

All4000ftvor 11 Monaten

Must have been 2 tired to continue.

Profilbild von Joshua S McConkey

Joshua S McConkeyvor 11 Monaten

@XiXiDu That is so far beyond anything else I have seen in robotics. What a huge step forward.

Profilbild von Porters Reserve

Porters Reservevor 11 Monaten

Great work now bring it to the reserve and let see if it can change the tires on our old tractor.

Profilbild von Ryan Industrialist

Ryan Industrialistvor 11 Monaten

33lb tire drag? Strong 'UPS driver tossing packages' vibes. Jokes aside, this hierarchical control could revolutionize high-speed depalletizing-finally bots that handle shifting pallets like humans.

Ähnliche Videos

Chinese humanoid robotics company LimX Dynamics has unveiled COSA (Cognitive Operating System of Agents). COSA is described as a unified "brain-body" architecture that allows the robot to think and act simultaneously in the real world. It integrates: - high-level cognition (reasoning, planning, adaptation) - and whole-body motion control (low-latency dynamic locomotion, manipulation) It enables the Oli robot to act on natural language instructions and perform tasks that involve both walking and manipulation while being adaptive to interruptions.

Chinese humanoid robotics company LimX Dynamics has unveiled COSA (Cognitive Operating System of Agents). COSA is described as a unified "brain-body" architecture that allows the robot to think and act simultaneously in the real world. It integrates: - high-level cognition (reasoning, planning, adaptation) - and whole-body motion control (low-latency dynamic locomotion, manipulation) It enables the Oli robot to act on natural language instructions and perform tasks that involve both walking and manipulation while being adaptive to interruptions.

The Humanoid Hub

71,168 Aufrufe • vor 8 Monaten

Noble Machines' Moby robot hauling a 50 lb crate. The whole-body control (WBC) is the physical foundation of the learning stack. It absorbs payload variation, contact response, and center-of-mass shifts in real time at the control layer and enabled cleaner data-focused learning

Noble Machines' Moby robot hauling a 50 lb crate. The whole-body control (WBC) is the physical foundation of the learning stack. It absorbs payload variation, contact response, and center-of-mass shifts in real time at the control layer and enabled cleaner data-focused learning

The Humanoid Hub

17,597 Aufrufe • vor 4 Monaten

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at

RAI Institute

13,560 Aufrufe • vor 1 Jahr

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.

The Humanoid Hub

91,980 Aufrufe • vor 1 Jahr

X-Humanoid just dropped TG-VLA — the first full-size, whole-body VLA framework for humanoids. 🤖 Most VLA demos today still look arm-centric: see, plan, reach, grasp. The legs are mostly there to carry the arms. TG-VLA pushes action into the whole body: → HEX: task context & cross-embodiment learning → HAF-VLA: high-DoF motion → structured action flows → DSRL-DCT: online RL through a compressed latent space The real shift isn't a smarter hand — it's locomotion, torso control, balance, and manipulation inside one control loop. That's the move: from mobile dual-arm machines to full-body agents.

X-Humanoid just dropped TG-VLA — the first full-size, whole-body VLA framework for humanoids. 🤖 Most VLA demos today still look arm-centric: see, plan, reach, grasp. The legs are mostly there to carry the arms. TG-VLA pushes action into the whole body: → HEX: task context & cross-embodiment learning → HAF-VLA: high-DoF motion → structured action flows → DSRL-DCT: online RL through a compressed latent space The real shift isn't a smarter hand — it's locomotion, torso control, balance, and manipulation inside one control loop. That's the move: from mobile dual-arm machines to full-body agents.

RoboHub🤖

42,167 Aufrufe • vor 3 Monaten

Just collecting manipulation data isn’t enough for robots - they need to be able to move around in the world, which has a whole different set of challenges from pure manipulation. And bringing navigation and manipulation together in a single framework is even more challenging. Enter HERMES, from Zhecheng Yuan and Tianming Wei. This is a four-stage process in which human videos are used to set up an RL sim-to-real training pipeline in order to overcome differences between robot and human kinematics, and used together with a navigation foundation model to move around in a variety of environments. To learn more, join us as Zhecheng Yuan and Tianming Wei tell us about how they built their system to perform mobile dexterous manipulation from human videos in a variety of environments. Watch Episode #45 of RoboPapers today, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

Just collecting manipulation data isn’t enough for robots - they need to be able to move around in the world, which has a whole different set of challenges from pure manipulation. And bringing navigation and manipulation together in a single framework is even more challenging. Enter HERMES, from Zhecheng Yuan and Tianming Wei. This is a four-stage process in which human videos are used to set up an RL sim-to-real training pipeline in order to overcome differences between robot and human kinematics, and used together with a navigation foundation model to move around in a variety of environments. To learn more, join us as Zhecheng Yuan and Tianming Wei tell us about how they built their system to perform mobile dexterous manipulation from human videos in a variety of environments. Watch Episode #45 of RoboPapers today, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

15,092 Aufrufe • vor 10 Monaten

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.

Space and Technology

31,172 Aufrufe • vor 15 Tagen

A new humanoid robot has been released. EstunCodroid, based in Nanjing, has unveiled its second-generation humanoid robot, Codroid2. Standing 170 cm tall and weighing 70 kg, Codroid2 features a payload capacity of 5 kg per arm and a total of 31 degrees of freedom. Powered by the VLA large language model, it supports multi-robot collaboration and is designed for applications in industrial manufacturing. It can perform physically demanding tasks such as component picking and material handling, as well as precision operations like screw fastening and quality inspection patrols, helping to reduce the burden on human workers. It is worth mentioning that EstunCodroid’s parent company, Estun, is a world-renowned industrial collaborative robot service provider.

A new humanoid robot has been released. EstunCodroid, based in Nanjing, has unveiled its second-generation humanoid robot, Codroid2. Standing 170 cm tall and weighing 70 kg, Codroid2 features a payload capacity of 5 kg per arm and a total of 31 degrees of freedom. Powered by the VLA large language model, it supports multi-robot collaboration and is designed for applications in industrial manufacturing. It can perform physically demanding tasks such as component picking and material handling, as well as precision operations like screw fastening and quality inspection patrols, helping to reduce the burden on human workers. It is worth mentioning that EstunCodroid’s parent company, Estun, is a world-renowned industrial collaborative robot service provider.

CyberRobo

30,114 Aufrufe • vor 1 Jahr

This is Clone Robotics realistic humanoid robotic arm, designed to closely mimic the movement and anatomy of a human arm. It is powered by electricity and a water-based hydraulic system using a compact pump about the size of a human heart. Despite weighing only 1 kilogram, the robotic arm can lift a 7-kilogram dumbbell.

This is Clone Robotics realistic humanoid robotic arm, designed to closely mimic the movement and anatomy of a human arm. It is powered by electricity and a water-based hydraulic system using a compact pump about the size of a human heart. Despite weighing only 1 kilogram, the robotic arm can lift a 7-kilogram dumbbell.

Space and Technology

58,940 Aufrufe • vor 3 Monaten

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵

Xiaolong Wang

159,462 Aufrufe • vor 2 Jahren

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!

RoboPapers

16,760 Aufrufe • vor 28 Tagen

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian ✈️ IROS join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian ✈️ IROS join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!

RoboPapers

30,095 Aufrufe • vor 4 Monaten

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

RoboPapers

27,360 Aufrufe • vor 5 Monaten

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,772 Aufrufe • vor 3 Monaten

A vibrant & independent civil society is essential to a healthy society. But the Kremlin weaponizes "civil society" and pseudo-academics to spread its propaganda in the West. Learn more about #Russia's use of #disinformation & information manipulation:

A vibrant & independent civil society is essential to a healthy society. But the Kremlin weaponizes "civil society" and pseudo-academics to spread its propaganda in the West. Learn more about #Russia's use of #disinformation & information manipulation:

Foreign Policy CAN

12,282 Aufrufe • vor 2 Jahren

NEW OPEN SOURCE ROBOT: A cheap, lightweight, and agile platform for locomotion! Called BRIDGE, it is an open-source humanoid robot platform, hardware plus control system. It is an 88 cm, 12.5 kg humanoid with 21 active degrees of freedom that costs roughly $1,500. Its distinguishing idea is that the body and the controller are designed together through a three-stage data-driven pipeline: - anatomical constraints - kinematic criteria - closed-loop dynamic criteria Hence the morphology itself is optimized to preserve human motion quality when the robot executes. It demonstrates teleoperated locomotion, balance-oriented motion (bending, lunging, extreme balance) and highly dynamic motion including back-flips and expressive sequences. I like that the thesis is designing the body and the controller together, not a controller for a fixed body, which inverts how nearly everyone else works; the robot is usually given, usually a Unitree G1, then train a policy on it. At the end of the day, it is a sub-$2K open-source humanoid that does back-flips, the humanoid analogue of the cheap-arm democratization.

NEW OPEN SOURCE ROBOT: A cheap, lightweight, and agile platform for locomotion! Called BRIDGE, it is an open-source humanoid robot platform, hardware plus control system. It is an 88 cm, 12.5 kg humanoid with 21 active degrees of freedom that costs roughly $1,500. Its distinguishing idea is that the body and the controller are designed together through a three-stage data-driven pipeline: - anatomical constraints - kinematic criteria - closed-loop dynamic criteria Hence the morphology itself is optimized to preserve human motion quality when the robot executes. It demonstrates teleoperated locomotion, balance-oriented motion (bending, lunging, extreme balance) and highly dynamic motion including back-flips and expressive sequences. I like that the thesis is designing the body and the controller together, not a controller for a fixed body, which inverts how nearly everyone else works; the robot is usually given, usually a Unitree G1, then train a policy on it. At the end of the day, it is a sub-$2K open-source humanoid that does back-flips, the humanoid analogue of the cheap-arm democratization.

Léo

23,248 Aufrufe • vor 13 Tagen

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 Aufrufe • vor 8 Monaten

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗

Xiaomeng Xu✈️IROS

21,329 Aufrufe • vor 3 Monaten