Загрузка видео...

Не удалось загрузить видео

На главную

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill...

DrJimFan's profile picture

Jim Fan

569,987 subscribers

209,534 просмотров • 2 месяцев назад •via X (Twitter)

ИгрыОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”
1:20

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”

Teslaconomics

42,563 просмотров • 6 месяцев назад

We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to announce DrEureka, an LLM agent that writes code to train robot skills in simulation, and writes more code to bridge the difficult simulation-reality gap. It fully automates the pipeline from new skill learning to real-world deployment. The Yoga ball task is particularly hard because it is not possible to accurately simulate the bouncy ball surface. Yet DrEureka has no trouble searching over a vast space of sim-to-real configurations, and enables the dog to steer the ball on various terrains, even walking sideways! Traditionally, the sim-to-real transfer is achieved by domain randomization, a tedious process that requires expert human roboticists to stare at every parameter and adjust by hand. Frontier LLMs like GPT-4 have tons of built-in physical intuition for friction, damping, stiffness, gravity, etc. We are (mildly) surprised to find that DrEureka can tune these parameters competently and explain its reasoning well. DrEureka builds on our prior work Eureka, the algorithm that teaches a 5-finger robot hand to do pen spinning. It takes one step further on our quest to automate the entire robot learning pipeline by an AI agent system. One model that outputs strings will supervise another model that outputs torque control. We open-source everything! Welcome you all to check out the paper, more videos, and try the codebase today: Code:
0:24

We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to announce DrEureka, an LLM agent that writes code to train robot skills in simulation, and writes more code to bridge the difficult simulation-reality gap. It fully automates the pipeline from new skill learning to real-world deployment. The Yoga ball task is particularly hard because it is not possible to accurately simulate the bouncy ball surface. Yet DrEureka has no trouble searching over a vast space of sim-to-real configurations, and enables the dog to steer the ball on various terrains, even walking sideways! Traditionally, the sim-to-real transfer is achieved by domain randomization, a tedious process that requires expert human roboticists to stare at every parameter and adjust by hand. Frontier LLMs like GPT-4 have tons of built-in physical intuition for friction, damping, stiffness, gravity, etc. We are (mildly) surprised to find that DrEureka can tune these parameters competently and explain its reasoning well. DrEureka builds on our prior work Eureka, the algorithm that teaches a 5-finger robot hand to do pen spinning. It takes one step further on our quest to automate the entire robot learning pipeline by an AI agent system. One model that outputs strings will supervise another model that outputs torque control. We open-source everything! Welcome you all to check out the paper, more videos, and try the codebase today: Code:

Jim Fan

909,065 просмотров • 2 лет назад

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk
1:39

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk

Mario Nawfal

19,289 просмотров • 6 месяцев назад

Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem. Partnering with Niantic Spatial 🌎 and NVIDIA Robotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user. The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments. #HumanoidRobots #Flexion #NianticSpatial #NVIDIA
1:21

Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem. Partnering with Niantic Spatial 🌎 and NVIDIA Robotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user. The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments. #HumanoidRobots #Flexion #NianticSpatial #NVIDIA

Flexion

71,981 просмотров • 1 месяц назад

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,960 просмотров • 9 месяцев назад

Every company needs killer Skills library. Not a folder in GitHub that 10% of the company knows how to use. A living library the whole company benefits from. The best deal review skill gets better every time a sales rep uses it. A great research skill gets discovered instead of rebuilt 50 times. Creating a skill should be as easy as asking to create something useful from based on the context of your work. And the top skills (and skill creators) should rise to the top. Create, share, collaborate on, discover and understand usage of skills, organize them into databases by team or use case, then use them across any of your agents. Still early, but the idea is to make Notion the best home for your company’s collective AI skills. Made a quick demo (all available now)!
1:59

Every company needs killer Skills library. Not a folder in GitHub that 10% of the company knows how to use. A living library the whole company benefits from. The best deal review skill gets better every time a sales rep uses it. A great research skill gets discovered instead of rebuilt 50 times. Creating a skill should be as easy as asking to create something useful from based on the context of your work. And the top skills (and skill creators) should rise to the top. Create, share, collaborate on, discover and understand usage of skills, organize them into databases by team or use case, then use them across any of your agents. Still early, but the idea is to make Notion the best home for your company’s collective AI skills. Made a quick demo (all available now)!

Hurley

691,328 просмотров • 12 дней назад

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵
0:58

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,246 просмотров • 1 год назад

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
53:45

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,772 просмотров • 2 месяцев назад

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.
1:01

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.

Praveen Kumar Verma

125,238 просмотров • 5 дней назад

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏
0:50

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 просмотров • 1 год назад

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:
0:47

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

22,872 просмотров • 10 месяцев назад

Elon Musk in new interview on how Tesla’s Optimus robot will learn more quickly: "What we're going to need to do is build a lot of robots and put them in a kind of like a "Optimus Academy" so they can do self-play in reality. We're actually building that out. We're gonna have at least 10,000 Optimus robots, maybe 20,000 or 30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator. A physics-accurate reality generator that we made for the cars. We'll do the same things for the robots, and actually have done that for the robots. You can do millions of simulated robots in a simulated world, and you use the tens of thousands in the real world to close the simulation to reality gap.”
0:59

Elon Musk in new interview on how Tesla’s Optimus robot will learn more quickly: "What we're going to need to do is build a lot of robots and put them in a kind of like a "Optimus Academy" so they can do self-play in reality. We're actually building that out. We're gonna have at least 10,000 Optimus robots, maybe 20,000 or 30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator. A physics-accurate reality generator that we made for the cars. We'll do the same things for the robots, and actually have done that for the robots. You can do millions of simulated robots in a simulated world, and you use the tens of thousands in the real world to close the simulation to reality gap.”

Sawyer Merritt

250,956 просмотров • 6 месяцев назад

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:
0:38

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Ilir Aliu

18,210 просмотров • 7 месяцев назад

Can United States manufacture robots? Matic Robots says "yes." It makes the best floor cleaning robot, that has won many perfect scores from Wired to many others. We love ours. But my trip there to get a tour from AI pioneer Navneet Dalal Navneet Dalal provided some real insights into how hard it is for a hardware company to make hardware in the United States. And how deeply AI is changing consumer electronics products that are going to be in many more homes soon. In this first part (Part II coming tomorrow) we get a look at how long it took for this company to go through prototypes to a shipping product. In the second part, you'll see the scaling hell that it takes to even ship a few thousand robots and the kinds of problems that scaling up a factory brings. Matic is one of my favorite small Silicon Valley companies. It has found what we call "product market fit." I just came back from CES where I saw many of its competitors, and the Matic wins because of not just the product thinking of Mehul and Navneet Dalal but because of their AI leadership. In a way their robot took many lessons from Tesla, from where to put the batteries to its bet on computer vision, which Navneet has been a pioneer in for years, working quietly behind the scenes. It is about to move into a new location that will allow it to grow to meet the demand that now is showing up (the boxes in its lobby show that it's outgrowing its current facilities). In terms of AI, it has aspirations of making a humanoid too, but it is taking a far more measured approach to getting there. By starting on the floor it can not just build world models based on real world data (customers are given a choice whether to allow its data to be used that way. Most customers choose to keep their data on the robot only, for privacy reasons, but if you opt in you can help them improve their models). They are using that data to understand homes. Navneet told me they hit very unusual situations in people's homes already that they couldn't really predict in simulators, like full-wall mirrors that confuse computer vision systems, or pools and water features in people's homes. Having real customers brings a ton of customer feedback about how to further improve the robot, and, as Navneet demonstrates in the second video, forces them to build a manufacturing muscle memory. Getting teams to work together, figuring out how to solve supply chain problems, from Trump's tarriffs, to a new one that showed up over the past couple of weeks. A supplier for its bags (one of the cheaper parts that goes into the robot) changed the glue it used, which caused robots to fail quality tests and the manufacturing line to stop. Reminds me a lot of the hell Elon Musk faced in its Fremont factory when Tesla was first starting to manufacture its Model 3, which almost bankrupted the company. Off the record Mehul and Navneet 🇮🇳 showed me some of the prototypes and plans for its next products that will show up over the next few years. Certainly not as sexy as Tesla, Figure, 1x_tech, and all the Chinese manufacturers are showing off already, but far better thought out for the typical Western home and AI plays a huge role in its future. It is the product that speaks for itself. It's amazing, and is about to get better this year due to AI. It's the first real vision-only robot to be in my home and I bet it won't be the last from this company. Real honor that they invited me over with my Insta360 camera (another company launched in my home, just like Matic was last year). In Part II we go into the factory.
14:24

Can United States manufacture robots? Matic Robots says "yes." It makes the best floor cleaning robot, that has won many perfect scores from Wired to many others. We love ours. But my trip there to get a tour from AI pioneer Navneet Dalal Navneet Dalal provided some real insights into how hard it is for a hardware company to make hardware in the United States. And how deeply AI is changing consumer electronics products that are going to be in many more homes soon. In this first part (Part II coming tomorrow) we get a look at how long it took for this company to go through prototypes to a shipping product. In the second part, you'll see the scaling hell that it takes to even ship a few thousand robots and the kinds of problems that scaling up a factory brings. Matic is one of my favorite small Silicon Valley companies. It has found what we call "product market fit." I just came back from CES where I saw many of its competitors, and the Matic wins because of not just the product thinking of Mehul and Navneet Dalal but because of their AI leadership. In a way their robot took many lessons from Tesla, from where to put the batteries to its bet on computer vision, which Navneet has been a pioneer in for years, working quietly behind the scenes. It is about to move into a new location that will allow it to grow to meet the demand that now is showing up (the boxes in its lobby show that it's outgrowing its current facilities). In terms of AI, it has aspirations of making a humanoid too, but it is taking a far more measured approach to getting there. By starting on the floor it can not just build world models based on real world data (customers are given a choice whether to allow its data to be used that way. Most customers choose to keep their data on the robot only, for privacy reasons, but if you opt in you can help them improve their models). They are using that data to understand homes. Navneet told me they hit very unusual situations in people's homes already that they couldn't really predict in simulators, like full-wall mirrors that confuse computer vision systems, or pools and water features in people's homes. Having real customers brings a ton of customer feedback about how to further improve the robot, and, as Navneet demonstrates in the second video, forces them to build a manufacturing muscle memory. Getting teams to work together, figuring out how to solve supply chain problems, from Trump's tarriffs, to a new one that showed up over the past couple of weeks. A supplier for its bags (one of the cheaper parts that goes into the robot) changed the glue it used, which caused robots to fail quality tests and the manufacturing line to stop. Reminds me a lot of the hell Elon Musk faced in its Fremont factory when Tesla was first starting to manufacture its Model 3, which almost bankrupted the company. Off the record Mehul and Navneet 🇮🇳 showed me some of the prototypes and plans for its next products that will show up over the next few years. Certainly not as sexy as Tesla, Figure, 1x_tech, and all the Chinese manufacturers are showing off already, but far better thought out for the typical Western home and AI plays a huge role in its future. It is the product that speaks for itself. It's amazing, and is about to get better this year due to AI. It's the first real vision-only robot to be in my home and I bet it won't be the last from this company. Real honor that they invited me over with my Insta360 camera (another company launched in my home, just like Matic was last year). In Part II we go into the factory.

Robert Scoble

69,229 просмотров • 7 месяцев назад

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:47

Robots might learn better from video than from language! 📼 Most Vision-Language-Action (VLA) models learn what to do from text, but still struggle with how things move in the real world. That makes them data-hungry and slow to train. mimic video takes a different route. Instead of grounding robot control in text, it grounds it in video, using large pre-trained video models that already capture physical motion and dynamics. The idea is straightforward: let the video model handle “what will happen next,” and let a smaller control model focus only on turning that visual plan into robot actions. The result is big gains in practice. Robots trained this way need 10× less data, converge twice as fast, and perform better on both simulated benchmarks and real bimanual manipulation tasks. If robots can “imagine” motion using video, control becomes a much simpler problem. Shoutout to Jonas Pai, Liam Achenbach, Oier Mees, Elvis Nava and the rest of the team! Here's the project page: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

50,090 просмотров • 8 месяцев назад

15–30 minutes of real-world robot data. That's now enough to go from sim-to-real failure to working robot. Let’s see… You train a robot in simulation. You deploy it in the real world. It fails. The physics don't match. So you try to fine-tune it with real data, but… you never have enough real data, and the fine-tuning breaks everything the simulation taught it. SimDist fixes this with one key decision: don't transfer the policy. Transfer the world model. Keep the reward and value knowledge from simulation frozen. Only update the part that's actually wrong, how the robot predicts physics. Now the robot doesn't have to relearn the entire task in the real world. It already knows what success looks like. It just needs to correct its understanding of how the real world moves. The part that makes this work: they also trained on failures and recoveries; not just perfect demonstrations. Without that, the planner finds the gaps and exploits them. With it, the robot can tell a good future from a bad one. That's all it needs. Results on peg insertion, table leg assembly, locomotion on slippery and uneven surfaces. Tasks that require precision, force, and quick reaction. Thanks for sharing, Tyler Westenbroek ([Tyler Westenbroek]. Interactive visualization + paper: ——- Weekly robotics and AI insights. Subscribe free:
0:41

15–30 minutes of real-world robot data. That's now enough to go from sim-to-real failure to working robot. Let’s see… You train a robot in simulation. You deploy it in the real world. It fails. The physics don't match. So you try to fine-tune it with real data, but… you never have enough real data, and the fine-tuning breaks everything the simulation taught it. SimDist fixes this with one key decision: don't transfer the policy. Transfer the world model. Keep the reward and value knowledge from simulation frozen. Only update the part that's actually wrong, how the robot predicts physics. Now the robot doesn't have to relearn the entire task in the real world. It already knows what success looks like. It just needs to correct its understanding of how the real world moves. The part that makes this work: they also trained on failures and recoveries; not just perfect demonstrations. Without that, the planner finds the gaps and exploits them. With it, the robot can tell a good future from a bad one. That's all it needs. Results on peg insertion, table leg assembly, locomotion on slippery and uneven surfaces. Tasks that require precision, force, and quick reaction. Thanks for sharing, Tyler Westenbroek ([Tyler Westenbroek]. Interactive visualization + paper: ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

14,222 просмотров • 3 месяцев назад

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake. Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence. ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. /goal: we all take a holiday and Jensen wouldn't even notice ;) We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:
1:15

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake. Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence. ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. /goal: we all take a holiday and Jensen wouldn't even notice ;) We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:

Jim Fan

675,804 просмотров • 2 месяцев назад

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.
0:50

We RL'ed humanoid robots to Cristiano Ronaldo, LeBron James, and Kobe Byrant! These are neural nets running on real hardware at our GEAR lab. Most robot demos you see online speed videos up. We actually *slow them down* so you can enjoy the fluid motions. I'm excited to announce "ASAP", a "real2sim2real" model that masters extremely smooth and dynamic motions for humanoid whole body control. We pretrain the robot in simulation first, but there is a notorious "sim2real" gap: it's very difficult for hand-engineered physics equations to match real world dynamics. Our fix is simple: just deploy a pretrained policy on real hardware, collect data, and replay the motion in sim. The replay will obviously have many errors, but that gives a rich signal to compensate for the physics discrepancy. Use another neural net to learn the delta. Basically, we "patch up" a traditional physics engine, so that the robot can experience almost the real world at scale in GPUs. The future is hybrid simulation: combine the power of classical sim engines refined over decades and the uncanny ability of modern NNs to capture a messy world.

Jim Fan

566,544 просмотров • 1 год назад

.Elon Musk told me he plans to build “Optimus Academy” to train an army of humanoid robots. > “millions of simulated robots in the simulated world” > “tens of thousands of robots in the real world, to close the simulation to reality gap.” Why? There’s two big differences between the way Tesla will train humanoid robots and the way it built Full Self-Driving. - FSD only requires learning three degrees of freedom: turning, accelerating, braking … and a humanoid robot has to learn how to coordinate 50+ joints. - You can sell a Tesla without FSD, which allows you to collect the millions and millions of hours of driving data needed to bootstrap your model ... but you can’t sell an Optimus that hasn’t learned how to do stuff yet. The Optimus Academy is Elon’s attempt to build the data flywheel that Tesla got for free with FSD.
2:10

.Elon Musk told me he plans to build “Optimus Academy” to train an army of humanoid robots. > “millions of simulated robots in the simulated world” > “tens of thousands of robots in the real world, to close the simulation to reality gap.” Why? There’s two big differences between the way Tesla will train humanoid robots and the way it built Full Self-Driving. - FSD only requires learning three degrees of freedom: turning, accelerating, braking … and a humanoid robot has to learn how to coordinate 50+ joints. - You can sell a Tesla without FSD, which allows you to collect the millions and millions of hours of driving data needed to bootstrap your model ... but you can’t sell an Optimus that hasn’t learned how to do stuff yet. The Optimus Academy is Elon’s attempt to build the data flywheel that Tesla got for free with FSD.

Dwarkesh Patel

551,640 просмотров • 6 месяцев назад

Our mainstream culture is conflict-avoidant. There is the assumption that the less conflict there is in a relationship, the better it is. I disagree. If there is real intimacy, conflict is actually unavoidable. Because we are all different. The real indicator of health in a relationship isn’t the lack of conflict, it is the quality of our repair. It is the level of authenticity we can bring. That requires knowing how to be with conflict and disagreement, how to hold nuance. And it also requires practicing the art of repairing. Are we willing to show up and repair what we broke? Do we have the skills and capacity required for that? Avoiding conflict means avoiding truth and therefore avoiding real intimacy. This work is essential and transcends the personal. If we aren't capable of dealing with conflict in our own personal lives, it is no wonder that the world is in the state it is, where the most prevalent conflict management strategy is escalation.
0:59

Our mainstream culture is conflict-avoidant. There is the assumption that the less conflict there is in a relationship, the better it is. I disagree. If there is real intimacy, conflict is actually unavoidable. Because we are all different. The real indicator of health in a relationship isn’t the lack of conflict, it is the quality of our repair. It is the level of authenticity we can bring. That requires knowing how to be with conflict and disagreement, how to hold nuance. And it also requires practicing the art of repairing. Are we willing to show up and repair what we broke? Do we have the skills and capacity required for that? Avoiding conflict means avoiding truth and therefore avoiding real intimacy. This work is essential and transcends the personal. If we aren't capable of dealing with conflict in our own personal lives, it is no wonder that the world is in the state it is, where the most prevalent conflict management strategy is escalation.

Laura Cincera

10,933 просмотров • 6 месяцев назад