Загрузка видео...

Не удалось загрузить видео

На главную

See Spot perform dynamic whole-body manipulation. Using a combination of reinforcement learning (RL) and sampling-based control, the robot is able to autonomously drag, roll, and stack tires weighing 15 kg (33 lb), well above its maximum arm lift capacity. Learn more about coordinating locomotion and manipulation processes:

rai_inst's profile picture

RAI Institute

7,885 subscribers

87,700 просмотров • 11 месяцев назад •via X (Twitter)

Наука и технологииОбразованиеСпорт
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 17

Фото профиля Flesh & Bone VR
Flesh & Bone VR11 месяцев назад

That leg up was super impressive. You can't teach these non humanoids with mocap.

Фото профиля Олег Сельдюгаев
Олег Сельдюгаев11 месяцев назад

This video makes it clear that a second hand is necessary for quick and precise loading and unloading operations. The second hand can be simpler than the primary hand.

Фото профиля iRiSh
iRiSh11 месяцев назад

I hope I live long enough to see the future where these are everywhere. I want one. Everyone wants one

Фото профиля Kwasikot
Kwasikot11 месяцев назад

The most amazing dog! What an incredible achievement! 😊

Фото профиля BadassRockets
BadassRockets11 месяцев назад

Two arms with guns Charging kennels Secured borders

Фото профиля Jo
Jo11 месяцев назад

Sandbag houses is the best use case. Whole houses built one bag at a time with a Spot bot.

Фото профиля Dennis
Dennis11 месяцев назад

Quadruped with one arm trying to train in novel ways to manipulate things that we already have plenty of human based training data, is rather inneficient.

Фото профиля Sammy
Sammy11 месяцев назад

It is doing it exceptionally well.

Фото профиля Max's Ghost
Max's Ghost11 месяцев назад

That's great. Come back when he can mount and balance it.

Фото профиля furtive pygmy (parody) 💨
furtive pygmy (parody) 💨11 месяцев назад

tires with ir markers on them are so futuristic, also narrow sidewalls

Фото профиля Mick Russom
Mick Russom11 месяцев назад

yeah, now tell people about the abysmal battery life.

Фото профиля Marie
Marie11 месяцев назад

Say NO to Robots! 🤖 Join the ARA.. fight for humanity! 😉

Фото профиля Anto Patrex
Anto Patrex11 месяцев назад

This is pretty impressive, specially with one arm.

Фото профиля All4000ft
All4000ft11 месяцев назад

Must have been 2 tired to continue.

Фото профиля Joshua S McConkey
Joshua S McConkey11 месяцев назад

@XiXiDu That is so far beyond anything else I have seen in robotics. What a huge step forward.

Фото профиля Porters Reserve
Porters Reserve11 месяцев назад

Great work now bring it to the reserve and let see if it can change the tires on our old tractor.

Фото профиля Ryan Industrialist
Ryan Industrialist11 месяцев назад

33lb tire drag? Strong 'UPS driver tossing packages' vibes. Jokes aside, this hierarchical control could revolutionize high-speed depalletizing-finally bots that handle shifting pallets like humans.

Похожие видео

Chinese humanoid robotics company LimX Dynamics has unveiled COSA (Cognitive Operating System of Agents). COSA is described as a unified "brain-body" architecture that allows the robot to think and act simultaneously in the real world. It integrates: - high-level cognition (reasoning, planning, adaptation) - and whole-body motion control (low-latency dynamic locomotion, manipulation) It enables the Oli robot to act on natural language instructions and perform tasks that involve both walking and manipulation while being adaptive to interruptions.
1:18

Chinese humanoid robotics company LimX Dynamics has unveiled COSA (Cognitive Operating System of Agents). COSA is described as a unified "brain-body" architecture that allows the robot to think and act simultaneously in the real world. It integrates: - high-level cognition (reasoning, planning, adaptation) - and whole-body motion control (low-latency dynamic locomotion, manipulation) It enables the Oli robot to act on natural language instructions and perform tasks that involve both walking and manipulation while being adaptive to interruptions.

The Humanoid Hub

71,168 просмотров • 8 месяцев назад

Noble Machines' Moby robot hauling a 50 lb crate. The whole-body control (WBC) is the physical foundation of the learning stack. It absorbs payload variation, contact response, and center-of-mass shifts in real time at the control layer and enabled cleaner data-focused learning
1:37

Noble Machines' Moby robot hauling a 50 lb crate. The whole-body control (WBC) is the physical foundation of the learning stack. It absorbs payload variation, contact response, and center-of-mass shifts in real time at the control layer and enabled cleaner data-focused learning

The Humanoid Hub

17,597 просмотров • 4 месяцев назад

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at
0:31

Researchers from RAI Institute present Diffuse-CLoC, a new control policy that fuses kinematic motion diffusion models with physics-based control to produce motions that are both physically realistic and precisely controllable. This breakthrough moves us closer to developing generalist policies that enable humanoid robots to perform diverse tasks, including dynamic locomotion and contact-rich manipulation, in a natural-looking and robust way. Learn more at

RAI Institute

13,560 просмотров • 1 год назад

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.
2:37

Humanoid robots playing table tennis fully autonomously. The 'HITTER' system combines a model-based planner with a reinforcement learning (RL) whole-body controller. It is fully autonomous but relies on an external sensing system. A 9-camera OptiTrack motion capture setup tracks the ball, which is covered with reflective markers, to predict its precise trajectory and compute the optimal strike position, velocity, and timing. The RL controller then executes coordinated arm, leg, and waist movements, producing agile, human-like returns.

The Humanoid Hub

91,980 просмотров • 1 год назад

X-Humanoid just dropped TG-VLA — the first full-size, whole-body VLA framework for humanoids. 🤖 Most VLA demos today still look arm-centric: see, plan, reach, grasp. The legs are mostly there to carry the arms. TG-VLA pushes action into the whole body: → HEX: task context & cross-embodiment learning → HAF-VLA: high-DoF motion → structured action flows → DSRL-DCT: online RL through a compressed latent space The real shift isn't a smarter hand — it's locomotion, torso control, balance, and manipulation inside one control loop. That's the move: from mobile dual-arm machines to full-body agents.
2:41

X-Humanoid just dropped TG-VLA — the first full-size, whole-body VLA framework for humanoids. 🤖 Most VLA demos today still look arm-centric: see, plan, reach, grasp. The legs are mostly there to carry the arms. TG-VLA pushes action into the whole body: → HEX: task context & cross-embodiment learning → HAF-VLA: high-DoF motion → structured action flows → DSRL-DCT: online RL through a compressed latent space The real shift isn't a smarter hand — it's locomotion, torso control, balance, and manipulation inside one control loop. That's the move: from mobile dual-arm machines to full-body agents.

RoboHub🤖

42,167 просмотров • 3 месяцев назад

Just collecting manipulation data isn’t enough for robots - they need to be able to move around in the world, which has a whole different set of challenges from pure manipulation. And bringing navigation and manipulation together in a single framework is even more challenging. Enter HERMES, from Zhecheng Yuan and Tianming Wei. This is a four-stage process in which human videos are used to set up an RL sim-to-real training pipeline in order to overcome differences between robot and human kinematics, and used together with a navigation foundation model to move around in a variety of environments. To learn more, join us as Zhecheng Yuan and Tianming Wei tell us about how they built their system to perform mobile dexterous manipulation from human videos in a variety of environments. Watch Episode #45 of RoboPapers today, hosted by Michael Cho - Rbt/Acc and Chris Paxton!
1:01:08

Just collecting manipulation data isn’t enough for robots - they need to be able to move around in the world, which has a whole different set of challenges from pure manipulation. And bringing navigation and manipulation together in a single framework is even more challenging. Enter HERMES, from Zhecheng Yuan and Tianming Wei. This is a four-stage process in which human videos are used to set up an RL sim-to-real training pipeline in order to overcome differences between robot and human kinematics, and used together with a navigation foundation model to move around in a variety of environments. To learn more, join us as Zhecheng Yuan and Tianming Wei tell us about how they built their system to perform mobile dexterous manipulation from human videos in a variety of environments. Watch Episode #45 of RoboPapers today, hosted by Michael Cho - Rbt/Acc and Chris Paxton!

RoboPapers

15,092 просмотров • 10 месяцев назад

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.
0:30

Chinese robotics company Unitree has developed a new AI model for general-purpose humanoid robots. The robot can autonomously sort objects, arrange flowers, stack cups, handle laundry, and organize shoes. The system combines vision, reasoning, and whole-body control to perform a wide range of household tasks.

Space and Technology

31,172 просмотров • 15 дней назад

A new humanoid robot has been released. EstunCodroid, based in Nanjing, has unveiled its second-generation humanoid robot, Codroid2. Standing 170 cm tall and weighing 70 kg, Codroid2 features a payload capacity of 5 kg per arm and a total of 31 degrees of freedom. Powered by the VLA large language model, it supports multi-robot collaboration and is designed for applications in industrial manufacturing. It can perform physically demanding tasks such as component picking and material handling, as well as precision operations like screw fastening and quality inspection patrols, helping to reduce the burden on human workers. It is worth mentioning that EstunCodroid’s parent company, Estun, is a world-renowned industrial collaborative robot service provider.
2:03

A new humanoid robot has been released. EstunCodroid, based in Nanjing, has unveiled its second-generation humanoid robot, Codroid2. Standing 170 cm tall and weighing 70 kg, Codroid2 features a payload capacity of 5 kg per arm and a total of 31 degrees of freedom. Powered by the VLA large language model, it supports multi-robot collaboration and is designed for applications in industrial manufacturing. It can perform physically demanding tasks such as component picking and material handling, as well as precision operations like screw fastening and quality inspection patrols, helping to reduce the burden on human workers. It is worth mentioning that EstunCodroid’s parent company, Estun, is a world-renowned industrial collaborative robot service provider.

CyberRobo

30,114 просмотров • 1 год назад

This is Clone Robotics realistic humanoid robotic arm, designed to closely mimic the movement and anatomy of a human arm. It is powered by electricity and a water-based hydraulic system using a compact pump about the size of a human heart. Despite weighing only 1 kilogram, the robotic arm can lift a 7-kilogram dumbbell.
0:24

This is Clone Robotics realistic humanoid robotic arm, designed to closely mimic the movement and anatomy of a human arm. It is powered by electricity and a water-based hydraulic system using a compact pump about the size of a human heart. Despite weighing only 1 kilogram, the robotic arm can lift a 7-kilogram dumbbell.

Space and Technology

58,940 просмотров • 3 месяцев назад

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵
3:15

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵

Xiaolong Wang

159,462 просмотров • 2 лет назад

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!
51:25

Reinforcement learning has unlocked tremendous gains in broader AI and humanoid robot whole body control, but still has yet to deliver strong results for robot manipulation due to the difficulty of capturing tasks in an accurate simulation; one way to fix this is to create complex environments with learned world models. But learning fully within world models is often intractable as powerful world models are computationally too expensive. Instead, Joseph Amigo and Rooholla Khorrambakht propose to break the problem down: into a large-scale world model that can generate forward trajectories, and a lightweight, low-dimensional latent-space model which can approximate local dynamics of a problem, without needing to back-propagate through a heavy global model. What this means: you can do reinforcement learning in this coupled local-global world model, and learn contact rich skills for a robot. To learn more, watch Episode #101 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan!

RoboPapers

16,760 просмотров • 28 дней назад

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian ✈️ IROS join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!
55:30

Sports like tennis are great examples of the sort of dynamic whole-body interaction that’s possible with humanoid robots. But capturing examples of fast, dynamic interactions from humans is really difficult. Enter LATENT, which uses lower-quality human data plus reinforcement learning to teach a robot to play tennis, able to complete back-and-forth volleys at a human level. LATENT has three steps: (1) collecting imperfect human data like a backswing, (2) using these to learn a latent action space, and (3) they train a high-level policy in simulation which can compose these actions and execute tennis skills on a robot. Haofei Lu and Yunrui Lian ✈️ IROS join us to tell us about their method. Watch Episode #80 of RoboPapers, with Chris Paxton and Jiafei Duan, now to learn more!

RoboPapers

30,095 просмотров • 4 месяцев назад

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!
44:20

Teaching robots to perform dexterous manipulation tasks currently requires teleoperation, which limits demonstration quality, speed, and scalability. Instead, why not use human videos? The problem is that a human hand isn’t a robot hand, so data must be retargeted using simulation to resolve issues like collisions and interpenetration when controlling the hand. In VideoManip, Hongyi Chen and co-authors built a system to solve this problem, taking in RGB videos of humans performing manipulation tasks and using them to create accurate simulations with which to learn robot policies. Watch episode #73 of RoboPapers, hosted by Michael Cho - Rbt/Acc and Chris Paxton, now to learn more!

RoboPapers

27,360 просмотров • 5 месяцев назад

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)
0:33

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 просмотров • 1 год назад

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
53:45

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,772 просмотров • 3 месяцев назад

A vibrant & independent civil society is essential to a healthy society. But the Kremlin weaponizes "civil society" and pseudo-academics to spread its propaganda in the West. Learn more about #Russia's use of #disinformation & information manipulation:
1:37

A vibrant & independent civil society is essential to a healthy society. But the Kremlin weaponizes "civil society" and pseudo-academics to spread its propaganda in the West. Learn more about #Russia's use of #disinformation & information manipulation:

Foreign Policy CAN

12,282 просмотров • 2 лет назад

NEW OPEN SOURCE ROBOT: A cheap, lightweight, and agile platform for locomotion! Called BRIDGE, it is an open-source humanoid robot platform, hardware plus control system. It is an 88 cm, 12.5 kg humanoid with 21 active degrees of freedom that costs roughly $1,500. Its distinguishing idea is that the body and the controller are designed together through a three-stage data-driven pipeline: - anatomical constraints - kinematic criteria - closed-loop dynamic criteria Hence the morphology itself is optimized to preserve human motion quality when the robot executes. It demonstrates teleoperated locomotion, balance-oriented motion (bending, lunging, extreme balance) and highly dynamic motion including back-flips and expressive sequences. I like that the thesis is designing the body and the controller together, not a controller for a fixed body, which inverts how nearly everyone else works; the robot is usually given, usually a Unitree G1, then train a policy on it. At the end of the day, it is a sub-$2K open-source humanoid that does back-flips, the humanoid analogue of the cheap-arm democratization.
0:45

NEW OPEN SOURCE ROBOT: A cheap, lightweight, and agile platform for locomotion! Called BRIDGE, it is an open-source humanoid robot platform, hardware plus control system. It is an 88 cm, 12.5 kg humanoid with 21 active degrees of freedom that costs roughly $1,500. Its distinguishing idea is that the body and the controller are designed together through a three-stage data-driven pipeline: - anatomical constraints - kinematic criteria - closed-loop dynamic criteria Hence the morphology itself is optimized to preserve human motion quality when the robot executes. It demonstrates teleoperated locomotion, balance-oriented motion (bending, lunging, extreme balance) and highly dynamic motion including back-flips and expressive sequences. I like that the thesis is designing the body and the controller together, not a controller for a fixed body, which inverts how nearly everyone else works; the robot is usually given, usually a Unitree G1, then train a policy on it. At the end of the day, it is a sub-$2K open-source humanoid that does back-flips, the humanoid analogue of the cheap-arm democratization.

Léo

23,248 просмотров • 13 дней назад

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla
1:06:07

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 просмотров • 8 месяцев назад

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗
0:24

🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗

Xiaomeng Xu✈️IROS

21,329 просмотров • 3 месяцев назад