Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

New framework: Kick down your robot, it will get back up every time 🥋 Chinese startup RoboParty is a Beijing startup founded April 2025 by Huang Yi, originally shipping ROBOTO ORIGIN, the world's first full-stack open-source bipedal humanoid. They released UFO: Unsupervised Reinforcement Learning Framework for Humanoid Control. DEFINITIONS... -> what differs is where the learning signal comes from: - SUPERVISED: humans supply the right answers (labels), the model imitates them. - UNSUPERVISED: no answer key, the model finds structure in raw data on its own. - REINFORCEMENT LEARNING: no answer key either, the model tries things and a reward scores each attempt. → UNSUPERVISED RL: trial and error where the agent invents its own rewards, instead of engineers hand-writing one per task. REPRESENTATION LEARNING: compress raw states into a useful internal map. TEMPORAL DISTANCE: distance on that map is "how many steps from A to B." CONTRASTIVE: trained by pulling together what's close in time, pushing apart what isn't. -> CONTRASTIVE TEMPORAL-DISTANCE REPRESENTATION LEARNING: the model builds an internal map of body states where distance means how many steps it takes to get from one to another. It is trained by contrast: states that occur close together in a movement get pulled together in the map, randomly paired states get pushed apart. UFO is an open-source training framework that teaches humanoid robots skills, like getting up, walking, goal-reaching, teleoperation, without reference motions -> no motion-capture or human-video demonstrations to imitate. Its core is TeCH, a contrastive temporal-distance representation-learning algorithm: the robot explores, builds pseudo-goals by temporal rolling, and learns goal-conditioned policies from a single unified progress reward. One framework trains five different robots (Unitree G1/H1, RoboParty RP0/RP1, AgiBot X2) with automatic config conversion in ~2–3 hours per robot! The real novelty here "no demonstrations at all". No data-collection arms race,the dominant humanoid-locomotion recipe is tracking: imitate mocap/retargeted-human reference trajectories. The robot self-generates goals from its own exploration and learns from a progress reward, needing zero reference motion data. Everybody else is fighting over data acquisition, while this team just teleports out of the race entirely (inb4 "competition is for losers 💀 ). This strategy reminds me of the DeepSeek playbook applied to robots: open-source the whole stack to become the global default and commoditize everyone else. RoboParty is giving away hardware and now control software (UFO) to be the Android of humanoids. Yet another reason for the US to ban Chinese open models perhaps 🥶 ? What I also really like about this approach is the cross-embodiment infrastructure, one framework trains Unitree G1/H1, RoboParty RP0/RP1, and AgiBot X2 with automatic configuration conversion. Just like Physical Intelligence, RoboParty seems to place itself as a neutral hardware agnostic middle man. Also woth mentioning: their ability ot perform stable skill injection, e.g. adding a cartwheel without forgetting how to walk. A common failure of RL humanoid policies is that teaching a new agile skill destabilizes the existing ones (catastrophic forgetting). UFO claims you can inject rare motions (cartwheel) without collapsing learned behavior. If it holds, that's a significant incremental/continual skill-learning! But again, I have to underline it: no arXiv, no external validation, no success-rate numbers. -> robotics badely needs an independent unbiased evaluator imho. Still, look at that cool demo: robot is getting kicked and pushed around (serious disturbance) during teleoperation (controlled the person at the back wearing the VR headset), and still managed to always get back up. This is some serious demonstration of stability and robustness!show more

Léo

1,016 subscribers

33,917 просмотров • 2 дней назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Open-source robot arm meets hand tracking [📍GitHub below] It is designed with an industrial mindset but built as a 3D-printed desktop system. PAROL6 paired with a LEAP Motion controller is a nice example of how accessible robot teleoperation has become. • Hand motion is streamed to the robot at 100 Hz via UDP • A pneumatic gripper is controlled by simple fist open and close gestures • The entire robot stack is open source, from mechanics to control software Combine that with low-latency hand tracking and you get a very practical platform for learning manipulation, teleoperation, and human-robot interfaces. This kind of setup is great for experimentation, teleop, data collection, and teaching robots by demonstration All without proprietary hardware or locked software. Credit to SourceRobotics 📍Code: —— Weekly robotics and AI insights. Subscribe free:

Open-source robot arm meets hand tracking [📍GitHub below] It is designed with an industrial mindset but built as a 3D-printed desktop system. PAROL6 paired with a LEAP Motion controller is a nice example of how accessible robot teleoperation has become. • Hand motion is streamed to the robot at 100 Hz via UDP • A pneumatic gripper is controlled by simple fist open and close gestures • The entire robot stack is open source, from mechanics to control software Combine that with low-latency hand tracking and you get a very practical platform for learning manipulation, teleoperation, and human-robot interfaces. This kind of setup is great for experimentation, teleop, data collection, and teaching robots by demonstration All without proprietary hardware or locked software. Credit to SourceRobotics 📍Code: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

32,025 просмотров • 6 месяцев назад

Experiments in progress. The one on the right has been learning for ~3 hours, the one in the middle for ~1 hour, and the one on the left just started a few minutes ago. The initial motivation for making the physical Atari was just to commit ourselves to a subset of algorithms that can make progress in this setup. This commitment rules out algorithms that require billions of samples to learn (or worse, require multiple environments running in parallel). Atari games are simple enough that we should be able to show learning on them in a short amount of time with no prior knowledge. Since then, I've realized that this setup is also a good way to compare different paradigms in robotics in a principled way. These paradigms are sim2real, learning from tele-operated data, and learning directly on the robots. So far, I have observed that getting sim2real to work reliably is hard. It requires tweaks that don't scale. Policies that can play perfectly in simulation fall apart because of latencies and the messiness of the real world. These aspects could be modeled to improve the simulation, but not without sinking significant human engineering hours. I have higher hopes for learning from tele-operated data, but that requires a human to learn the task first. These experiments are on my to-do list. I have to learn to play some of the games well through the robot. I’m half-decent at playing Pong and Ms Pacman now. Learning directly on robots is looking like the most promising approach. This approach takes away pesky distribution shifts and makes it possible to have algorithms that continually improve with more data and time without any human intervention. It feels great to let experiments run overnight and wake up to find improved policies. With learning on robots, I should, in principle, be able to go on a long vacation and come back to find better policies for complex tasks beyond Atari games. Whether that is possible with current learning algorithms is a different question.

Experiments in progress. The one on the right has been learning for ~3 hours, the one in the middle for ~1 hour, and the one on the left just started a few minutes ago. The initial motivation for making the physical Atari was just to commit ourselves to a subset of algorithms that can make progress in this setup. This commitment rules out algorithms that require billions of samples to learn (or worse, require multiple environments running in parallel). Atari games are simple enough that we should be able to show learning on them in a short amount of time with no prior knowledge. Since then, I've realized that this setup is also a good way to compare different paradigms in robotics in a principled way. These paradigms are sim2real, learning from tele-operated data, and learning directly on the robots. So far, I have observed that getting sim2real to work reliably is hard. It requires tweaks that don't scale. Policies that can play perfectly in simulation fall apart because of latencies and the messiness of the real world. These aspects could be modeled to improve the simulation, but not without sinking significant human engineering hours. I have higher hopes for learning from tele-operated data, but that requires a human to learn the task first. These experiments are on my to-do list. I have to learn to play some of the games well through the robot. I’m half-decent at playing Pong and Ms Pacman now. Learning directly on robots is looking like the most promising approach. This approach takes away pesky distribution shifts and makes it possible to have algorithms that continually improve with more data and time without any human intervention. It feels great to let experiments run overnight and wake up to find improved policies. With learning on robots, I should, in principle, be able to go on a long vacation and come back to find better policies for complex tasks beyond Atari games. Whether that is possible with current learning algorithms is a different question.

Khurram Javed

52,110 просмотров • 8 месяцев назад

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

Rohan Paul

37,121 просмотров • 5 месяцев назад

Twitter boi, do you want to see a humanoid AI lab from the inside? Well I do ;) Join me to visit Flexion A world-leading lab building the AI brains for humanoids – in Zurich, one of the best hubs for Physical AI. Their goal? Build the Android of robotics. One operating system that works on any humanoid hardware. It is crazy what their robots can do. We pushed a robot on the stairs. It didn't care. They left one in a rainy forest it was never trained on. Chilled and hiked there. And because they let robots train on their own, from zero through reinforcement learning, the robots do odd inhuman things. Eg they stand up from the ground in ways no human ever would. If their humanoid brain works, this will be industry defining. This is why Flexion is one of Europe's Most Ambitious Startups. 🇪🇺🔥 0:00 Intro 0:38 Inside a Lab with 14 Humanoids 2:17 What’s Actually Inside a Humanoid? 4:43 The Robot Navigates on Its Own 6:31 Why Grasping Is So Hard 7:01 Stairs, Pushes, and Balance Tests 7:55 Training 4,000 Robots in Parallel 10:50 Teleoperation vs Reinforcement Learning 14:35 The Case for Humanoids 15:55 Why Europe Could Win Robotics Please like & RT 🙏

Twitter boi, do you want to see a humanoid AI lab from the inside? Well I do ;) Join me to visit Flexion A world-leading lab building the AI brains for humanoids – in Zurich, one of the best hubs for Physical AI. Their goal? Build the Android of robotics. One operating system that works on any humanoid hardware. It is crazy what their robots can do. We pushed a robot on the stairs. It didn't care. They left one in a rainy forest it was never trained on. Chilled and hiked there. And because they let robots train on their own, from zero through reinforcement learning, the robots do odd inhuman things. Eg they stand up from the ground in ways no human ever would. If their humanoid brain works, this will be industry defining. This is why Flexion is one of Europe's Most Ambitious Startups. 🇪🇺🔥 0:00 Intro 0:38 Inside a Lab with 14 Humanoids 2:17 What’s Actually Inside a Humanoid? 4:43 The Robot Navigates on Its Own 6:31 Why Grasping Is So Hard 7:01 Stairs, Pushes, and Balance Tests 7:55 Training 4,000 Robots in Parallel 10:50 Teleoperation vs Reinforcement Learning 14:35 The Case for Humanoids 15:55 Why Europe Could Win Robotics Please like & RT 🙏

Andreas Klinger 🦾

14,761 просмотров • 4 месяцев назад

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 просмотров • 1 год назад

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. "Trained model" is a repo of sensorimotor skills instead of floating weights. “Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches. Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;) Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! Deep dive in thread:

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library. ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent. "Trained model" is a repo of sensorimotor skills instead of floating weights. “Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches. Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of training compute ;) Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours! Deep dive in thread:

Jim Fan

204,520 просмотров • 1 месяц назад

Sub-1 hour? Chasing the human half-marathon record? That’s suddenly the conversation around Beijing’s humanoid robot race this Sunday. 🏃‍♂️🤖 A year ago, this event was mostly about finishing. Now the bar is speed, stability, and full-course autonomy. More than 100 teams and 300+ humanoid robots are expected on the line, and nearly 40% of them are entering the autonomous navigation group instead of relying on remote control. The course is not getting easier either. Same 21.0975 km, but with more urban slopes, rolling sections, and park terrain built to expose weak points in control, perception, and endurance. That is what makes this race worth watching. If some of these machines really get close to one hour, the story is no longer “robots can run.” It becomes “humanoids are starting to hold speed, balance, and navigation together over real distance.” And that matters far beyond a race course. This is one of the clearest public tests yet for where humanoid robotics actually stands in hardware, motion control, power systems, and autonomous mobility. Sunday’s headline will be the time. The bigger takeaway will be how much of that pace is still there at kilometer 15, 18, and 21. I want to see who is actually fast, who is truly autonomous, and who can finish without falling apart. That is where the real signal will be.

Sub-1 hour? Chasing the human half-marathon record? That’s suddenly the conversation around Beijing’s humanoid robot race this Sunday. 🏃‍♂️🤖 A year ago, this event was mostly about finishing. Now the bar is speed, stability, and full-course autonomy. More than 100 teams and 300+ humanoid robots are expected on the line, and nearly 40% of them are entering the autonomous navigation group instead of relying on remote control. The course is not getting easier either. Same 21.0975 km, but with more urban slopes, rolling sections, and park terrain built to expose weak points in control, perception, and endurance. That is what makes this race worth watching. If some of these machines really get close to one hour, the story is no longer “robots can run.” It becomes “humanoids are starting to hold speed, balance, and navigation together over real distance.” And that matters far beyond a race course. This is one of the clearest public tests yet for where humanoid robotics actually stands in hardware, motion control, power systems, and autonomous mobility. Sunday’s headline will be the time. The bigger takeaway will be how much of that pace is still there at kilometer 15, 18, and 21. I want to see who is actually fast, who is truly autonomous, and who can finish without falling apart. That is where the real signal will be.

RoboHub🤖

209,428 просмотров • 3 месяцев назад

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

Pascal Bornet

186,223 просмотров • 3 месяцев назад

Not a preplanned motion sequence. A robot deciding mid-jump what to do next. [📍 paper + demo] Researchers just showed a humanoid doing real parkour using only onboard perception. No motion script, no fixed obstacle layout. The system is called Perceptive Humanoid Parkour (PHP). Instead of memorizing a path, the robot reads depth from its cameras and continuously chooses actions. Step, vault, climb, or roll depending on what geometry appears in front of it. To make that possible, they combine three ideas: First, they stitch together human motion clips into long movement references so the robot learns fluid transitions instead of isolated tricks. Second, they train tracking policies with reinforcement learning so contacts land at the right time and the robot keeps balance during dynamic moves. Finally, everything is distilled into one perception policy that runs directly from depth input to action selection. The result on a Unitree G1: about 3 m/s vaults wall climbs up to 1.25 m nearly one minute continuous obstacle traversal adapting when obstacles move What matters is not the tricks. It is the shift in capability. Earlier humanoids executed motions. This one navigates situations. Once robots react to geometry instead of replaying trajectories, environments stop needing to be predictable. Warehouses, homes, and outdoors suddenly become the same problem. Thanks for sharing, Zhen Wu! Paper + demo: ——— Weekly robotics and AI insights. Subscribe free:

Not a preplanned motion sequence. A robot deciding mid-jump what to do next. [📍 paper + demo] Researchers just showed a humanoid doing real parkour using only onboard perception. No motion script, no fixed obstacle layout. The system is called Perceptive Humanoid Parkour (PHP). Instead of memorizing a path, the robot reads depth from its cameras and continuously chooses actions. Step, vault, climb, or roll depending on what geometry appears in front of it. To make that possible, they combine three ideas: First, they stitch together human motion clips into long movement references so the robot learns fluid transitions instead of isolated tricks. Second, they train tracking policies with reinforcement learning so contacts land at the right time and the robot keeps balance during dynamic moves. Finally, everything is distilled into one perception policy that runs directly from depth input to action selection. The result on a Unitree G1: about 3 m/s vaults wall climbs up to 1.25 m nearly one minute continuous obstacle traversal adapting when obstacles move What matters is not the tricks. It is the shift in capability. Earlier humanoids executed motions. This one navigates situations. Once robots react to geometry instead of replaying trajectories, environments stop needing to be predictable. Warehouses, homes, and outdoors suddenly become the same problem. Thanks for sharing, Zhen Wu! Paper + demo: ——— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

22,080 просмотров • 5 месяцев назад

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

This is how ALOHA's "teleoperation" system works - a fancy word for "remote control". Training robots will be more and more like playing games in the physical world. A human operates a "joystick++" to perform tasks and collect data, or intervene if there's any safety concern. There's actually a learning curve to master the controller, much like practicing gaming skills. Teleoperation can be done in many different ways. ALOHA is an impressive custom-built system with very low cost. Here're a few alternatives: (1) Motion Capture (MoCap): apply the MoCap systems used for Hollywood movies to capture the fine-grained motions of hand joints. There would be no "embodiment gap" if the robot hand has 5 fingers. For instance, a demonstrator can wear a CyberGlove ( and manipulate the objects. CyberGlove will capture the motion signals & haptic feedback in real-time, which can be re-targeted onto the humanoid. (2) Wearing gloves & markers can be clumsy. An alternative way to do MoCap is through computer vision. DexPilot from NVIDIA enables marker-less and glove-free data collection. The human operator simply uses their bare hands to perform the tasks. 4 Intel RealSense depth cameras and 2 NVIDIA Titan XP GPUs (yeah, 2019 work) translate the pixels to precise motion signals for robot learning. (3) VR Headset: turn the training room into a VR game and "role play" the robot. This has the advantage of scalable remote data collection - annotators from around the world can contribute without coming onsite. VR demonstration technique appeared in research projects like the iGibson home robot simulator, an initiative that I participated in at Stanford: Behind-the-scene video by Litian Liang

Jim Fan

124,588 просмотров • 2 лет назад

Can United States manufacture robots? Matic Robots says "yes." It makes the best floor cleaning robot, that has won many perfect scores from Wired to many others. We love ours. But my trip there to get a tour from AI pioneer Navneet Dalal Navneet Dalal provided some real insights into how hard it is for a hardware company to make hardware in the United States. And how deeply AI is changing consumer electronics products that are going to be in many more homes soon. In this first part (Part II coming tomorrow) we get a look at how long it took for this company to go through prototypes to a shipping product. In the second part, you'll see the scaling hell that it takes to even ship a few thousand robots and the kinds of problems that scaling up a factory brings. Matic is one of my favorite small Silicon Valley companies. It has found what we call "product market fit." I just came back from CES where I saw many of its competitors, and the Matic wins because of not just the product thinking of Mehul and Navneet Dalal but because of their AI leadership. In a way their robot took many lessons from Tesla, from where to put the batteries to its bet on computer vision, which Navneet has been a pioneer in for years, working quietly behind the scenes. It is about to move into a new location that will allow it to grow to meet the demand that now is showing up (the boxes in its lobby show that it's outgrowing its current facilities). In terms of AI, it has aspirations of making a humanoid too, but it is taking a far more measured approach to getting there. By starting on the floor it can not just build world models based on real world data (customers are given a choice whether to allow its data to be used that way. Most customers choose to keep their data on the robot only, for privacy reasons, but if you opt in you can help them improve their models). They are using that data to understand homes. Navneet told me they hit very unusual situations in people's homes already that they couldn't really predict in simulators, like full-wall mirrors that confuse computer vision systems, or pools and water features in people's homes. Having real customers brings a ton of customer feedback about how to further improve the robot, and, as Navneet demonstrates in the second video, forces them to build a manufacturing muscle memory. Getting teams to work together, figuring out how to solve supply chain problems, from Trump's tarriffs, to a new one that showed up over the past couple of weeks. A supplier for its bags (one of the cheaper parts that goes into the robot) changed the glue it used, which caused robots to fail quality tests and the manufacturing line to stop. Reminds me a lot of the hell Elon Musk faced in its Fremont factory when Tesla was first starting to manufacture its Model 3, which almost bankrupted the company. Off the record Mehul and Navneet 🇮🇳 showed me some of the prototypes and plans for its next products that will show up over the next few years. Certainly not as sexy as Tesla, Figure, 1x_tech, and all the Chinese manufacturers are showing off already, but far better thought out for the typical Western home and AI plays a huge role in its future. It is the product that speaks for itself. It's amazing, and is about to get better this year due to AI. It's the first real vision-only robot to be in my home and I bet it won't be the last from this company. Real honor that they invited me over with my Insta360 camera (another company launched in my home, just like Matic was last year). In Part II we go into the factory.

Can United States manufacture robots? Matic Robots says "yes." It makes the best floor cleaning robot, that has won many perfect scores from Wired to many others. We love ours. But my trip there to get a tour from AI pioneer Navneet Dalal Navneet Dalal provided some real insights into how hard it is for a hardware company to make hardware in the United States. And how deeply AI is changing consumer electronics products that are going to be in many more homes soon. In this first part (Part II coming tomorrow) we get a look at how long it took for this company to go through prototypes to a shipping product. In the second part, you'll see the scaling hell that it takes to even ship a few thousand robots and the kinds of problems that scaling up a factory brings. Matic is one of my favorite small Silicon Valley companies. It has found what we call "product market fit." I just came back from CES where I saw many of its competitors, and the Matic wins because of not just the product thinking of Mehul and Navneet Dalal but because of their AI leadership. In a way their robot took many lessons from Tesla, from where to put the batteries to its bet on computer vision, which Navneet has been a pioneer in for years, working quietly behind the scenes. It is about to move into a new location that will allow it to grow to meet the demand that now is showing up (the boxes in its lobby show that it's outgrowing its current facilities). In terms of AI, it has aspirations of making a humanoid too, but it is taking a far more measured approach to getting there. By starting on the floor it can not just build world models based on real world data (customers are given a choice whether to allow its data to be used that way. Most customers choose to keep their data on the robot only, for privacy reasons, but if you opt in you can help them improve their models). They are using that data to understand homes. Navneet told me they hit very unusual situations in people's homes already that they couldn't really predict in simulators, like full-wall mirrors that confuse computer vision systems, or pools and water features in people's homes. Having real customers brings a ton of customer feedback about how to further improve the robot, and, as Navneet demonstrates in the second video, forces them to build a manufacturing muscle memory. Getting teams to work together, figuring out how to solve supply chain problems, from Trump's tarriffs, to a new one that showed up over the past couple of weeks. A supplier for its bags (one of the cheaper parts that goes into the robot) changed the glue it used, which caused robots to fail quality tests and the manufacturing line to stop. Reminds me a lot of the hell Elon Musk faced in its Fremont factory when Tesla was first starting to manufacture its Model 3, which almost bankrupted the company. Off the record Mehul and Navneet 🇮🇳 showed me some of the prototypes and plans for its next products that will show up over the next few years. Certainly not as sexy as Tesla, Figure, 1x_tech, and all the Chinese manufacturers are showing off already, but far better thought out for the typical Western home and AI plays a huge role in its future. It is the product that speaks for itself. It's amazing, and is about to get better this year due to AI. It's the first real vision-only robot to be in my home and I bet it won't be the last from this company. Real honor that they invited me over with my Insta360 camera (another company launched in my home, just like Matic was last year). In Part II we go into the factory.

Robert Scoble

69,229 просмотров • 6 месяцев назад

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Excited to announce GR00T N1, the world’s first open foundation model for humanoid robots! We are on a mission to democratize Physical AI. The power of general robot brain, in the palm of your hand - with only 2B parameters, N1 learns from the most diverse physical action dataset ever compiled and punches above its weight: - Real humanoid teleoperation data. - Large-scale simulation data: we are open-sourcing 300K+ trajectories! - Neural trajectories: we apply SOTA video generation models to “hallucinate” new synthetic data that features accurate physics in pixels. Using Jensen’s words, “systematically infinite data”! - Latent actions: we develop novel algorithms to extract action tokens from in-the-wild human videos and neural generated videos. GR00T N1 is a single end-to-end neural net, from photons to actions: - Vision-Language Model (System 2) that interprets the physical world through vision and language instructions, enabling robots to reason about their environment and instructions, and plan the right actions. - Diffusion Transformer (System 1) that “renders” smooth and precise motor actions at 120 Hz, executing the latent plan made by System 2. We deploy N1 on GR1 robot, 1X Neo robot, and a large collection of simulation benchmarks. N1 achieves up to +30% boost in diverse manipulation tasks for household and industrial settings. While humanoid robots are the main focus of N1, our model also supports cross-embodiment. We finetune it to work on the $110 HuggingFace LeRobot SO100 robot arm! Open robot brain runs on open hardware. Sounds just right. Let’s solve robotics, together, one token at a time. Links to our Whitepaper, Github repo, HuggingFace model, and open dataset page in the thread: 🧵

Jim Fan

466,333 просмотров • 1 год назад

this is the world's first ever humanoid robot that will summit Mt. Everest it's named Pemba, and two days ago it reached the top of Chimborazo, a 20,000-foot peak in Ecuador, completely on its own (no remote or operator). and the robot itself is nothing special. Pemba is a Unitree G1, the same ~$14k robot anyone can buy online right now. the guy behind it, Pablo Berlanga, is doing this to send robots into the places that kill people. think about how you'd check on a melting glacier or a deadly crevasse out in the middle of nowhere today. you either send a person who might not come back, or you skip it and learn nothing. so Berlanga wants a robot that walks in on its own, carries a few pounds of gear, and brings back footage from places no human can safely reach. i think eventually they'll even be used for robot rescue missions to reach people stranded in disasters and dangerous situations from here, the plan for Pemba is Mauna Kea in Hawaii, then Everest. Everest is the tricky one. the team wanted to send Pemba up this spring to start hauling trash off the mountain and tracking its glaciers (something Nepal genuinely needs help with) but the Nepalese government told them to wait. there's no law in Nepal for a climber that isn't human, so the rules have to get written before a robot can set foot on the mountain. kind of incredible that the machine is ready for Everest a full year before anyone's decided whether it's allowed to be up there

this is the world's first ever humanoid robot that will summit Mt. Everest it's named Pemba, and two days ago it reached the top of Chimborazo, a 20,000-foot peak in Ecuador, completely on its own (no remote or operator). and the robot itself is nothing special. Pemba is a Unitree G1, the same ~$14k robot anyone can buy online right now. the guy behind it, Pablo Berlanga, is doing this to send robots into the places that kill people. think about how you'd check on a melting glacier or a deadly crevasse out in the middle of nowhere today. you either send a person who might not come back, or you skip it and learn nothing. so Berlanga wants a robot that walks in on its own, carries a few pounds of gear, and brings back footage from places no human can safely reach. i think eventually they'll even be used for robot rescue missions to reach people stranded in disasters and dangerous situations from here, the plan for Pemba is Mauna Kea in Hawaii, then Everest. Everest is the tricky one. the team wanted to send Pemba up this spring to start hauling trash off the mountain and tracking its glaciers (something Nepal genuinely needs help with) but the Nepalese government told them to wait. there's no law in Nepal for a climber that isn't human, so the rules have to get written before a robot can set foot on the mountain. kind of incredible that the machine is ready for Everest a full year before anyone's decided whether it's allowed to be up there

Ole Lehmann

33,490 просмотров • 1 месяц назад

New short course on Reinforcement Learning from Human Feedback! RLHF is one of the key techniques that led to the rise of modern LLMs. It is used to align LLMs with human preferences, to make them more honest, helpful and harmless, by (i) learning a reward function that mimics human preferences, as expressed in human-provided labels, then, (ii) tuning an LLM to generate outputs that receive a high reward. In this course, taught by Nikita Namjoshi, Developer Advocate for GenAI at Google Cloud, you'll learn the details of how RLHF works, including how to apply it to tune an LLM for your own applications. You'll also use an open source library to tune a base LLM to align with human preferences expressed in a training set, and evaluate the tuned model by comparing its responses before and after RLHF-tuning. Please sign up here!

New short course on Reinforcement Learning from Human Feedback! RLHF is one of the key techniques that led to the rise of modern LLMs. It is used to align LLMs with human preferences, to make them more honest, helpful and harmless, by (i) learning a reward function that mimics human preferences, as expressed in human-provided labels, then, (ii) tuning an LLM to generate outputs that receive a high reward. In this course, taught by Nikita Namjoshi, Developer Advocate for GenAI at Google Cloud, you'll learn the details of how RLHF works, including how to apply it to tune an LLM for your own applications. You'll also use an open source library to tune a base LLM to align with human preferences expressed in a training set, and evaluate the tuned model by comparing its responses before and after RLHF-tuning. Please sign up here!

Andrew Ng

205,542 просмотров • 2 лет назад

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Let's reverse engineer this demo. You need 3 things: (1) robust hardware and motor designs that treat simulation as first-class citizen; (2) a human motion capture ("mocap") dataset, such as those for film and gaming characters; (3) massively parallel RL training in GPU-accelerated simulation. Last October, our team trained a 1.5M parameter foundation model called HOVER for such agile motor control. It follows this recipe, roughly speaking (details in thread): (1) Simulation used to be an after-thought. Now, it has to be part of the hardware design process. If your robot doesn't simulate well, you can kiss RL goodbye. Hardware-simulation co-design is a very interesting emergent topic that only becomes meaningful with today's compute capability. (2) Human mocap dataset to produce natural-looking walking and running gaits. That's one huge advantage of using humanoid robot - you get to imitate from tons of human motions that were originally captured for movies or AAA games. At least 3 ways to use the data: - For initialization: pre-train the neural net to imitate human, and then finetune it into the robot form factor with physics turned on; - For reward function: penalize any deviations from the target pose; - For representation learning: treat the human poses as a "motion prior" to constrain the space of robot behaviors. (3) Shove the above into Isaac sim, add a lot of randomization, pump it through PPO, throw in a bunch of GPUs, and then watch Netflix till loss converges. If you have an urge to comment this is CGI, let me save you a few keystrokes — many academic labs now own the G1 robot in the flesh. See our team's HOVER work in the thread: 🧵

Jim Fan

216,139 просмотров • 1 год назад

Dr Fei-Fei-Li explains with a simple example how everyday household chores are so extremely difficult for Robots. "If you tell a robot to open the top drawer and watch out for the vase, this is actually a really hard task for robots." because the robot must ground language into the real world. Words like "top", "drawer", and "vase" are abstract. The system has to map them to 3D locations, objects, and relations in a noisy scene. This requires robust perception, object recognition, and spatial reasoning under uncertainty. The robot also lacks human commonsense. "Watch out" implies predicting consequences, estimating clearances, and understanding that vases are fragile. Encoding such priors, like how heavy a drawer is or how a vase might tip, is very complex and difficult without rich world knowledge. Learning the behavior from rewards is tough. The success signal is very sparse here, so naive exploration almost never stumbles on a full success sequence. This makes policy learning sample inefficient and brittle, especially when the environment changes between training and deployment. A sparse reward situation is when the agent only gets a success signal at the very end, and gets little or no feedback along the way. If a robot must open a drawer without hitting a vase, it might get reward only if the drawer ends up open and the vase is intact. Every partial try before that looks the same to the learner, reward equals 0. --- From "DSAI by Dr. Osbert Tay" YT channel

Dr Fei-Fei-Li explains with a simple example how everyday household chores are so extremely difficult for Robots. "If you tell a robot to open the top drawer and watch out for the vase, this is actually a really hard task for robots." because the robot must ground language into the real world. Words like "top", "drawer", and "vase" are abstract. The system has to map them to 3D locations, objects, and relations in a noisy scene. This requires robust perception, object recognition, and spatial reasoning under uncertainty. The robot also lacks human commonsense. "Watch out" implies predicting consequences, estimating clearances, and understanding that vases are fragile. Encoding such priors, like how heavy a drawer is or how a vase might tip, is very complex and difficult without rich world knowledge. Learning the behavior from rewards is tough. The success signal is very sparse here, so naive exploration almost never stumbles on a full success sequence. This makes policy learning sample inefficient and brittle, especially when the environment changes between training and deployment. A sparse reward situation is when the agent only gets a success signal at the very end, and gets little or no feedback along the way. If a robot must open a drawer without hitting a vase, it might get reward only if the drawer ends up open and the vase is intact. Every partial try before that looks the same to the learner, reward equals 0. --- From "DSAI by Dr. Osbert Tay" YT channel

Rohan Paul

342,603 просмотров • 8 месяцев назад

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

Today, we're joined by Nikita Rudin, co-founder and CEO of Flexion to discuss the gap between current robotic capabilities and what’s required to deploy fully autonomous robots in the real world. Nikita explains how reinforcement learning and simulation have driven rapid progress in robot locomotion—and why locomotion is still far from “solved.” We dig into the sim2real gap, and how adding visual inputs introduces noise and significantly complicates sim-to-real transfer. We also explore the debate between end-to-end models and modular approaches, and why separating locomotion, planning, and semantics remains a pragmatic approach today. Nikita also introduces the concept of "real-to-sim", which uses real-world data to refine simulation parameters for higher fidelity training, discusses how reinforcement learning, imitation learning, and teleoperation data are combined to train robust policies for both quadruped and humanoid robots, and introduces Flexion's hierarchical approach that utilizes pre-trained Vision-Language Models (VLMs) for high-level task orchestration with Vision-Language-Action (VLA) models and low-level whole-body trackers. Finally, Nikita shares the behind-the-scenes in humanoid robot demos, his take on reinforcement learning in simulation versus the real world, the nuances of reward tuning, and offers practical advice for researchers and practitioners looking to get started in robotics today. 🗒️ For the full list of resources for this episode, visit the show notes page: 📖 CHAPTERS =============================== 00:00 - Introduction 04:07 - Is robot locomotion solved? 06:04 - Sim-to-real gap 08:58 - Adding semantics to policies 09:42 - Modular vs end-to-end architectures 10:29 - Planner model 12:21 - Adapting RL techniques from quadrupeds to humanoids 15:39 - Behind robot demos 18:09 - Humanoid robots in home environments 22:03 - Training approach 23:56 - VLA models 27:59 - Closing the sim-to-real gap 32:55 - Task orchestration using VLMs 36:38 - Tool use 38:10 - Model hierarchy 43:37 - Simulator versus simulation environment 44:57 - Combining imitation learning and reinforcement learning 46:42 - RL in real world versus RL in simulation 52:58 - Reward tuning and value functions in robotics 56:38 - Predictions 1:00:10 - Humanoids, quadropeds, and wheeled platforms 1:02:45 - Advice, recommended robot kits, and community pla

The TWIML AI Podcast

22,592 просмотров • 6 месяцев назад

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data. 2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the motion of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro -> RoboCasa produces N (varying visuals) -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are building tools to enable everyone in the ecosystem to scale up with us. Links in thread:

Jim Fan

364,380 просмотров • 2 лет назад

𝗖𝗵𝗶𝗻𝗮 𝗶𝘀 𝗳𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝘁𝗵𝗲 𝗵𝘂𝗺𝗮𝗻𝗼𝗶𝗱 𝗿𝗼𝗯𝗼𝘁 𝗿𝗮𝗰𝗲 𝗯𝗲𝗳𝗼𝗿𝗲 𝗺𝗼𝘀𝘁 𝗼𝗳 𝘁𝗵𝗲 𝗪𝗲𝘀𝘁 𝗿𝗲𝗮𝗹𝗶𝘇𝗲𝘀 𝗶𝘁 𝗵𝗮𝘀 𝘀𝘁𝗮𝗿𝘁𝗲𝗱. AGIBOT held its Partner Conference in Shanghai last week. The real headline wasn't the new hardware. It was their CTO standing on stage, telling investors that humanoid R&D season is over. 2026, he said, is "Deployment Year One." Not research. Not demos. Deployment into real factories, real warehouses, real stores. The manufacturing ramp is getting faster. 1,000 humanoid robots in the first 2 years. Another 4,000 in the next 12 months. Another 5,000 in just 3 months after that. AGIBOT is now shipping more humanoids per quarter than most US robotics companies have built in their entire existence. Then came the announcements the industry will spend the rest of the year reacting to. AIMA. The first full-stack open architecture for embodied AI. A unified robot operating system called Link-U, three dev platforms for motion, interaction, and task creation, plus an open agent framework. Any developer can build on top of it. This is the Android play for humanoids. GO-2. A vision-language-action foundation model with Action Chain-of-Thought reasoning. Planning and execution collapsed into one model. GE-2. A world model for simulation, strategy testing, and sim-to-real transfer. AGIBOT WORLD 2026. An open-source, production-grade real-world dataset pulled from actual industrial, logistics, hotel, and commercial sites. Seven standardized "productivity packages" covering logistics sorting, retail service, security patrol, commercial cleaning, and more. Plug, deploy, bill. A 5-year, $280 million commitment to seed a global developer and partner ecosystem. Now look at the competition. Boston Dynamics has been building humanoids since 1992. Tesla's Optimus is still climbing its own hype curve. Apptronik and Agility are well-funded but pre-scale on real deployments. AGIBOT has pulled all of this off in three years, with no acquisitions, no legacy platform, and no IPO distractions. While the West is still asking when humanoids will scale, China is already shipping them by the thousand.

𝗖𝗵𝗶𝗻𝗮 𝗶𝘀 𝗳𝗶𝗻𝗶𝘀𝗵𝗶𝗻𝗴 𝘁𝗵𝗲 𝗵𝘂𝗺𝗮𝗻𝗼𝗶𝗱 𝗿𝗼𝗯𝗼𝘁 𝗿𝗮𝗰𝗲 𝗯𝗲𝗳𝗼𝗿𝗲 𝗺𝗼𝘀𝘁 𝗼𝗳 𝘁𝗵𝗲 𝗪𝗲𝘀𝘁 𝗿𝗲𝗮𝗹𝗶𝘇𝗲𝘀 𝗶𝘁 𝗵𝗮𝘀 𝘀𝘁𝗮𝗿𝘁𝗲𝗱. AGIBOT held its Partner Conference in Shanghai last week. The real headline wasn't the new hardware. It was their CTO standing on stage, telling investors that humanoid R&D season is over. 2026, he said, is "Deployment Year One." Not research. Not demos. Deployment into real factories, real warehouses, real stores. The manufacturing ramp is getting faster. 1,000 humanoid robots in the first 2 years. Another 4,000 in the next 12 months. Another 5,000 in just 3 months after that. AGIBOT is now shipping more humanoids per quarter than most US robotics companies have built in their entire existence. Then came the announcements the industry will spend the rest of the year reacting to. AIMA. The first full-stack open architecture for embodied AI. A unified robot operating system called Link-U, three dev platforms for motion, interaction, and task creation, plus an open agent framework. Any developer can build on top of it. This is the Android play for humanoids. GO-2. A vision-language-action foundation model with Action Chain-of-Thought reasoning. Planning and execution collapsed into one model. GE-2. A world model for simulation, strategy testing, and sim-to-real transfer. AGIBOT WORLD 2026. An open-source, production-grade real-world dataset pulled from actual industrial, logistics, hotel, and commercial sites. Seven standardized "productivity packages" covering logistics sorting, retail service, security patrol, commercial cleaning, and more. Plug, deploy, bill. A 5-year, $280 million commitment to seed a global developer and partner ecosystem. Now look at the competition. Boston Dynamics has been building humanoids since 1992. Tesla's Optimus is still climbing its own hype curve. Apptronik and Agility are well-funded but pre-scale on real deployments. AGIBOT has pulled all of this off in three years, with no acquisitions, no legacy platform, and no IPO distractions. While the West is still asking when humanoids will scale, China is already shipping them by the thousand.

Shruti

214,939 просмотров • 3 месяцев назад

The most interesting part for me is where Andrej Karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.” A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer. > “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.” So what do humans do instead? > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” > “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.” Why can’t we just add this training to LLMs today? > “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.” > “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.” > “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.” How do humans get around model collapse? > “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.” In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by Erik Hoel. I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization? > “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”

The most interesting part for me is where Andrej Karpathy describes why LLMs aren't able to learn like humans. As you would expect, he comes up with a wonderfully evocative phrase to describe RL: “sucking supervision bits through a straw.” A single end reward gets broadcast across every token in a successful trajectory, upweighting even wrong or irrelevant turns that lead to the right answer. > “Humans don't use reinforcement learning, as I've said before. I think they do something different. Reinforcement learning is a lot worse than the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse.” So what do humans do instead? > “The book I’m reading is a set of prompts for me to do synthetic data generation. It's by manipulating that information that you actually gain that knowledge. We have no equivalent of that with LLMs; they don't really do that.” > “I'd love to see during pretraining some kind of a stage where the model thinks through the material and tries to reconcile it with what it already knows. There's no equivalent of any of this. This is all research.” Why can’t we just add this training to LLMs today? > “There are very subtle, hard to understand reasons why it's not trivial. If I just give synthetic generation of the model thinking about a book, you look at it and you're like, 'This looks great. Why can't I train on it?' You could try, but the model will actually get much worse if you continue trying.” > “Say we have a chapter of a book and I ask an LLM to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.” > “You're not getting the richness and the diversity and the entropy from these models as you would get from humans. How do you get synthetic data generation to work despite the collapse and while maintaining the entropy? It is a research problem.” How do humans get around model collapse? > “These analogies are surprisingly good. Humans collapse during the course of their lives. Children haven't overfit yet. They will say stuff that will shock you. Because they're not yet collapsed. But we [adults] are collapsed. We end up revisiting the same thoughts, we end up saying more and more of the same stuff, the learning rates go down, the collapse continues to get worse, and then everything deteriorates.” In fact, there’s an interesting paper arguing that dreaming evolved to assist generalization, and resist overfitting to daily learning - look up The Overfitted Brain by Erik Hoel. I asked Karpathy: Isn’t it interesting that humans learn best at a part of their lives (childhood) whose actual details they completely forget, adults still learn really well but have terrible memory about the particulars of the things they read or watch, and LLMs can memorize arbitrary details about text that no human could but are currently pretty bad at generalization? > “[Fallible human memory] is a feature, not a bug, because it forces you to only learn the generalizable components. LLMs are distracted by all the memory that they have of the pre-trained documents. That's why when I talk about the cognitive core, I actually want to remove the memory. I'd love to have them have less memory so that they have to look things up and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue for acting.”

Dwarkesh Patel

1,051,605 просмотров • 9 месяцев назад