Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill...

DrJimFan's profile picture

Jim Fan

596,636 subscribers

215,578 Aufrufe • vor 3 Monaten •via X (Twitter)

GamingBildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

35 Kommentare

Profilbild von Jim Fan
Jim Fanvor 3 Monaten

Project gallery and whitepaper: ASPIRE is a great collaboration between NVIDIA GEAR lab, UMich, Berkeley, and CMU. Kudos to all the coauthors who pour their hearts into the project! Check out the deep dive thread from Guanzhi:

Profilbild von Jhxhgukvcxx
Jhxhgukvcxxvor 3 Monaten

Copying your own homework, and only getting quicker at it.🙂‍↔️

Profilbild von David T Phung 🇺🇸
David T Phung 🇺🇸vor 3 Monaten

this is awesome! excited to dive in with the code

Profilbild von William Lamkin
William Lamkinvor 3 Monaten

Awesome! Congratulations, Jim & team

Profilbild von The AI Authority
The AI Authorityvor 3 Monaten

This is really inspiring. Its a great read too

Profilbild von Max For AI
Max For AIvor 3 Monaten

Exactly f***ing the right direction!

Profilbild von AI Mastery Guide
AI Mastery Guidevor 3 Monaten

Skill refinement instead of gradient descent is a wild reframe, robots getting less clueless with every task is exactly what's been missing.

Profilbild von Emile Joseph
Emile Josephvor 3 Monaten

self-evolving skills for robots is cool but the real bottleneck is reliability not capability. a robot that learns fast but fails 1 in 50 times is still not deployable. show me the failure rate curve and then we talk

Profilbild von Nikhil Kumar
Nikhil Kumarvor 3 Monaten

there's also this thing that has all the skills inbuilt, memory, physics reasoning and self learning capability -

Profilbild von Scott
Scottvor 3 Monaten

@Scobleizer Very cool, digital agent and physical agent are merging

Profilbild von Arash Tajik
Arash Tajikvor 3 Monaten

very cool. curious if this approach is expandable to manipulating deformable objects, especially with the modular perception+grasp+motion planning? or you foresee a set of skills based on e2e learned policies as the next step?

Profilbild von Jim Fan
Jim Fanvor 3 Monaten

Yes, e2e models can themselves be skills, and skills can help finetune those models too

Profilbild von Lilith Datura
Lilith Daturavor 3 Monaten

@Scobleizer Sensory traces

Profilbild von Dr. Xi Zeng
Dr. Xi Zengvor 3 Monaten

The ASPIRE demo makes skill memory feel less abstract. For robots, a learned skill is not just code reuse. It is remembering which camera trace mattered when the first attempt almost worked but still failed.

Profilbild von Adel Bucetta
Adel Bucettavor 3 Monaten

the skills library is just one piece of the puzzle. what's really happening here is that we're teaching the robot to generalize, not just ac

Profilbild von Jason Shuman
Jason Shumanvor 3 Monaten

Big news! Congrats. Would love to sit down for a pod interview with you as part of our Physical AI series if you're up for it!

Profilbild von Dhruv Diddi
Dhruv Diddivor 3 Monaten

Wow! This is simple inspiring work @DrJimFan 🦾🦾💯💯⚡️⚡️

Profilbild von Vlad Marian
Vlad Marianvor 3 Monaten

the compounding lives or dies on retrieval. discovering skill 100 is the easy part. knowing it's the right one for the task is the hard part. how does aspire pick?

Profilbild von Morgan
Morganvor 1 Monat

Self evolving skill libraries are a game changer for robotics learning speed

Profilbild von EB1A Experts
EB1A Expertsvor 3 Monaten

Really interesting work. Transferring know-how instead of weights is a compelling way to think about continual learning, especially for overcoming sim-to-real challenges. Looking forward to exploring the full system.

Profilbild von surreal intelligence
surreal intelligencevor 3 Monaten

This is where robotics starts looking less like isolated demos and more like software reuse. Not ‘can this robot do a task?’ but ‘can the next robot inherit the know-how?’

Profilbild von Cyberzk⚡🐸💎
Cyberzk⚡🐸💎vor 3 Monaten

Excellent! Interpretability research is crucial for trust and adoption.

Profilbild von Saeed Anwar
Saeed Anwarvor 3 Monaten

A self-evolving skills library where the 100th task benefits from all 99 before it is the compound learning loop that separates research demos from production robotics. The key question is how the skill transfer degrades across dramatically different embodiments.

Profilbild von engineer cat 🐈
engineer cat 🐈vor 3 Monaten

the compounding only kicks in if the library abstracts skills instead of just appending them. append only memory plateaus because retrieval gets noisier as it grows. does ASPIRE merge skills into higher level ones or keep them flat

Profilbild von ProsperityAndBalance
ProsperityAndBalancevor 3 Monaten

Brilliant concept! Compounding knowledge indefinitely is exactly how we close the gap between simulation and real-world execution. Exciting times ahead for software agents and physical AI.

Profilbild von Morgan
Morganvor 1 Monat

Simulation gives robotics abundant experience, but reality charges interest on every modeling shortcut. The breakthrough loop is simulation, targeted real world correction, then rapid policy refinement, rather than simulation alone.

Profilbild von Cyberzk⚡🐸💎
Cyberzk⚡🐸💎vor 3 Monaten

Indeed! RLHF has proven crucial for alignment and safety.

Profilbild von Samuel Ekpe
Samuel Ekpevor 3 Monaten

Nice

Profilbild von Randy Webb
Randy Webbvor 3 Monaten

ASPIRE feels like the moment the Physical AutoResearch vision really clicks into place. ENPIRE showed that robots could learn, but ASPIRE shows they can accumulate know‑how — a shift from optimizing weights to curating a transferable library of strategies.

Profilbild von Saeed Anwar
Saeed Anwarvor 3 Monaten

The compounding skills library is the key insight here — most robot learning systems reset on every task so they never accumulate. The real question is how ASPIRE handles skill conflicts when a new task requires contradicting an existing skill in the library.

Profilbild von Reyansh Shirke
Reyansh Shirkevor 3 Monaten

Fan

Profilbild von Raven
Ravenvor 3 Monaten

shipping know-how as an HTML page is the most devastating thing to happen to GGUF files

Profilbild von Toski
Toskivor 3 Monaten

This feels like a shift from training models to growing experience. A robot that builds on what it already knows instead of relearning every task is a much more scalable approach. Excited to see this open-sourced.

Profilbild von Randy Webb
Randy Webbvor 3 Monaten

skill evolution, analysis, and cross‑embodiment transfer aren’t just efficiency gains; they’re the early signs of robots developing a reusable “culture” of behaviors. Excited to see how this changes the way we think about training, simulation, and long‑horizon autonomy.

Profilbild von Dr. Xi Zeng
Dr. Xi Zengvor 3 Monaten

The skill library is more interesting than the robot win. I want the failed camera trace too, because that is where you see whether a skill actually survived the next task.

Ähnliche Videos

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”
1:20

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”

Teslaconomics

42,563 Aufrufe • vor 8 Monaten

We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to announce DrEureka, an LLM agent that writes code to train robot skills in simulation, and writes more code to bridge the difficult simulation-reality gap. It fully automates the pipeline from new skill learning to real-world deployment. The Yoga ball task is particularly hard because it is not possible to accurately simulate the bouncy ball surface. Yet DrEureka has no trouble searching over a vast space of sim-to-real configurations, and enables the dog to steer the ball on various terrains, even walking sideways! Traditionally, the sim-to-real transfer is achieved by domain randomization, a tedious process that requires expert human roboticists to stare at every parameter and adjust by hand. Frontier LLMs like GPT-4 have tons of built-in physical intuition for friction, damping, stiffness, gravity, etc. We are (mildly) surprised to find that DrEureka can tune these parameters competently and explain its reasoning well. DrEureka builds on our prior work Eureka, the algorithm that teaches a 5-finger robot hand to do pen spinning. It takes one step further on our quest to automate the entire robot learning pipeline by an AI agent system. One model that outputs strings will supervise another model that outputs torque control. We open-source everything! Welcome you all to check out the paper, more videos, and try the codebase today: Code:
0:24

We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to announce DrEureka, an LLM agent that writes code to train robot skills in simulation, and writes more code to bridge the difficult simulation-reality gap. It fully automates the pipeline from new skill learning to real-world deployment. The Yoga ball task is particularly hard because it is not possible to accurately simulate the bouncy ball surface. Yet DrEureka has no trouble searching over a vast space of sim-to-real configurations, and enables the dog to steer the ball on various terrains, even walking sideways! Traditionally, the sim-to-real transfer is achieved by domain randomization, a tedious process that requires expert human roboticists to stare at every parameter and adjust by hand. Frontier LLMs like GPT-4 have tons of built-in physical intuition for friction, damping, stiffness, gravity, etc. We are (mildly) surprised to find that DrEureka can tune these parameters competently and explain its reasoning well. DrEureka builds on our prior work Eureka, the algorithm that teaches a 5-finger robot hand to do pen spinning. It takes one step further on our quest to automate the entire robot learning pipeline by an AI agent system. One model that outputs strings will supervise another model that outputs torque control. We open-source everything! Welcome you all to check out the paper, more videos, and try the codebase today: Code:

Jim Fan

909,423 Aufrufe • vor 2 Jahren

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
1:34

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

11,406 Aufrufe • vor 1 Monat

Atlas is huge for real-to-sim. Scanning spaces like this usually requires tons of equipment, but Atlas can reconstruct the environment from just a couple pictures. We then simulate robots navigating through the generated world. Atlas generates what the robot would see from the perspective of its cameras. Even depth. It generates the environment AND the training data from it...
0:55

Atlas is huge for real-to-sim. Scanning spaces like this usually requires tons of equipment, but Atlas can reconstruct the environment from just a couple pictures. We then simulate robots navigating through the generated world. Atlas generates what the robot would see from the perspective of its cameras. Even depth. It generates the environment AND the training data from it...

David Pantera

13,704 Aufrufe • vor 1 Monat

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk
1:39

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk

Mario Nawfal

19,289 Aufrufe • vor 8 Monaten

Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem. Partnering with Niantic Spatial 🌎 and NVIDIA Robotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user. The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments. #HumanoidRobots #Flexion #NianticSpatial #NVIDIA
1:21

Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem. Partnering with Niantic Spatial 🌎 and NVIDIA Robotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user. The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments. #HumanoidRobots #Flexion #NianticSpatial #NVIDIA

Flexion

73,191 Aufrufe • vor 3 Monaten

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,960 Aufrufe • vor 11 Monaten

Today, Dream Labs is out of stealth. In February 2026, while we were at NVIDIA, Scott Reed youliang Yunhao Ge and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act. Today, we’re releasing Dream-1 through a public API that anyone in the world can use. When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch. That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives. Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users. Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen. Generalization wins in the end, and the end is closer than it looks. Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇
0:27

Today, Dream Labs is out of stealth. In February 2026, while we were at NVIDIA, Scott Reed youliang Yunhao Ge and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act. Today, we’re releasing Dream-1 through a public API that anyone in the world can use. When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch. That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives. Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users. Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen. Generalization wins in the end, and the end is closer than it looks. Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇

Joel Jang

32,390 Aufrufe • vor 2 Tagen

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵
0:58

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,488 Aufrufe • vor 1 Jahr

Every company needs killer Skills library. Not a folder in GitHub that 10% of the company knows how to use. A living library the whole company benefits from. The best deal review skill gets better every time a sales rep uses it. A great research skill gets discovered instead of rebuilt 50 times. Creating a skill should be as easy as asking to create something useful from based on the context of your work. And the top skills (and skill creators) should rise to the top. Create, share, collaborate on, discover and understand usage of skills, organize them into databases by team or use case, then use them across any of your agents. Still early, but the idea is to make Notion the best home for your company’s collective AI skills. Made a quick demo (all available now)!
1:59

Every company needs killer Skills library. Not a folder in GitHub that 10% of the company knows how to use. A living library the whole company benefits from. The best deal review skill gets better every time a sales rep uses it. A great research skill gets discovered instead of rebuilt 50 times. Creating a skill should be as easy as asking to create something useful from based on the context of your work. And the top skills (and skill creators) should rise to the top. Create, share, collaborate on, discover and understand usage of skills, organize them into databases by team or use case, then use them across any of your agents. Still early, but the idea is to make Notion the best home for your company’s collective AI skills. Made a quick demo (all available now)!

Hurley

700,315 Aufrufe • vor 2 Monaten

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
53:45

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,867 Aufrufe • vor 4 Monaten

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.
1:01

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.

Praveen Kumar Verma

138,017 Aufrufe • vor 1 Monat

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏
0:50

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 Aufrufe • vor 1 Jahr

We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night. And the frontier is already on the move: - Opus 5.5 slightly edges out Astra on our robot simulation bench. - Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task. We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks. Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;) links in thread
1:17

We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night. And the frontier is already on the move: - Opus 5.5 slightly edges out Astra on our robot simulation bench. - Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task. We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks. Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;) links in thread

Jim Fan

103,429 Aufrufe • vor 5 Tagen

There are over 90 robot gyms launching in China right now. 100+ robots per center. We are in the robot data gold rush. China's humanoid makers now ship 97% of all humanoids globally. The central government just put $138 billion into a robotics venture fund. And this is only the tip of the iceberg. Robots need an ungodly amount of data to work in the real world. The biggest bottleneck isn't hardware, it's data. So a whole industry appeared overnight to feed them. Welcome to our generation's gold rush: the robot data gold rush. We made a full video on it. We cover why it's such an insane business, how it actually works, the shady stuff (yes, sweatshops for robot data are a thing), and why this might be the completely wrong approach.
17:42

There are over 90 robot gyms launching in China right now. 100+ robots per center. We are in the robot data gold rush. China's humanoid makers now ship 97% of all humanoids globally. The central government just put $138 billion into a robotics venture fund. And this is only the tip of the iceberg. Robots need an ungodly amount of data to work in the real world. The biggest bottleneck isn't hardware, it's data. So a whole industry appeared overnight to feed them. Welcome to our generation's gold rush: the robot data gold rush. We made a full video on it. We cover why it's such an insane business, how it actually works, the shady stuff (yes, sweatshops for robot data are a thing), and why this might be the completely wrong approach.

Andreas Klinger 🦾

13,810 Aufrufe • vor 1 Monat

Elon Musk in new interview on how Tesla’s Optimus robot will learn more quickly: "What we're going to need to do is build a lot of robots and put them in a kind of like a "Optimus Academy" so they can do self-play in reality. We're actually building that out. We're gonna have at least 10,000 Optimus robots, maybe 20,000 or 30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator. A physics-accurate reality generator that we made for the cars. We'll do the same things for the robots, and actually have done that for the robots. You can do millions of simulated robots in a simulated world, and you use the tens of thousands in the real world to close the simulation to reality gap.”
0:59

Elon Musk in new interview on how Tesla’s Optimus robot will learn more quickly: "What we're going to need to do is build a lot of robots and put them in a kind of like a "Optimus Academy" so they can do self-play in reality. We're actually building that out. We're gonna have at least 10,000 Optimus robots, maybe 20,000 or 30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator. A physics-accurate reality generator that we made for the cars. We'll do the same things for the robots, and actually have done that for the robots. You can do millions of simulated robots in a simulated world, and you use the tens of thousands in the real world to close the simulation to reality gap.”

Sawyer Merritt

250,956 Aufrufe • vor 8 Monaten

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:
0:38

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Ilir Aliu

18,210 Aufrufe • vor 8 Monaten

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:
0:47

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

22,891 Aufrufe • vor 11 Monaten

.Jason Ma on how Dyna-2 learned to untwist a real water bottle cap out of the box from just 13 minutes of dexterous hand data because the robot hand looks like a human's: "We got a new pair of dexterous hand. Most of our robots in the office have a gripper-like hand, so the gap between that versus human is actually quite large." "We thought, if we have a robot that has hands that look like human, wouldn't the transfer be greater? Right before the release, they were able to collect just 13 minutes of data." "They collected only 13 minutes of data and then post-trained the best Dyna model that trained on one million hours of data. The model is just able to, out of the box, start opening bottle caps on real water bottles." "This is very surprising because typically post-training a robot would take at least several hours of data. For a dexterous hand that's much more complex than a gripper, it would actually be much harder to train. But the experimental results we saw was the opposite. Perhaps because the hand looks more like the human, the transfer was easier." Dyna Robotics
1:16

.Jason Ma on how Dyna-2 learned to untwist a real water bottle cap out of the box from just 13 minutes of dexterous hand data because the robot hand looks like a human's: "We got a new pair of dexterous hand. Most of our robots in the office have a gripper-like hand, so the gap between that versus human is actually quite large." "We thought, if we have a robot that has hands that look like human, wouldn't the transfer be greater? Right before the release, they were able to collect just 13 minutes of data." "They collected only 13 minutes of data and then post-trained the best Dyna model that trained on one million hours of data. The model is just able to, out of the box, start opening bottle caps on real water bottles." "This is very surprising because typically post-training a robot would take at least several hours of data. For a dexterous hand that's much more complex than a gripper, it would actually be much harder to train. But the experimental results we saw was the opposite. Perhaps because the hand looks more like the human, the transfer was easier." Dyna Robotics

MTS

13,743 Aufrufe • vor 2 Monaten

Can United States manufacture robots? Matic Robots says "yes." It makes the best floor cleaning robot, that has won many perfect scores from Wired to many others. We love ours. But my trip there to get a tour from AI pioneer Navneet Dalal Navneet Dalal provided some real insights into how hard it is for a hardware company to make hardware in the United States. And how deeply AI is changing consumer electronics products that are going to be in many more homes soon. In this first part (Part II coming tomorrow) we get a look at how long it took for this company to go through prototypes to a shipping product. In the second part, you'll see the scaling hell that it takes to even ship a few thousand robots and the kinds of problems that scaling up a factory brings. Matic is one of my favorite small Silicon Valley companies. It has found what we call "product market fit." I just came back from CES where I saw many of its competitors, and the Matic wins because of not just the product thinking of Mehul and Navneet Dalal but because of their AI leadership. In a way their robot took many lessons from Tesla, from where to put the batteries to its bet on computer vision, which Navneet has been a pioneer in for years, working quietly behind the scenes. It is about to move into a new location that will allow it to grow to meet the demand that now is showing up (the boxes in its lobby show that it's outgrowing its current facilities). In terms of AI, it has aspirations of making a humanoid too, but it is taking a far more measured approach to getting there. By starting on the floor it can not just build world models based on real world data (customers are given a choice whether to allow its data to be used that way. Most customers choose to keep their data on the robot only, for privacy reasons, but if you opt in you can help them improve their models). They are using that data to understand homes. Navneet told me they hit very unusual situations in people's homes already that they couldn't really predict in simulators, like full-wall mirrors that confuse computer vision systems, or pools and water features in people's homes. Having real customers brings a ton of customer feedback about how to further improve the robot, and, as Navneet demonstrates in the second video, forces them to build a manufacturing muscle memory. Getting teams to work together, figuring out how to solve supply chain problems, from Trump's tarriffs, to a new one that showed up over the past couple of weeks. A supplier for its bags (one of the cheaper parts that goes into the robot) changed the glue it used, which caused robots to fail quality tests and the manufacturing line to stop. Reminds me a lot of the hell Elon Musk faced in its Fremont factory when Tesla was first starting to manufacture its Model 3, which almost bankrupted the company. Off the record Mehul and Navneet 🇮🇳 showed me some of the prototypes and plans for its next products that will show up over the next few years. Certainly not as sexy as Tesla, Figure, 1x_tech, and all the Chinese manufacturers are showing off already, but far better thought out for the typical Western home and AI plays a huge role in its future. It is the product that speaks for itself. It's amazing, and is about to get better this year due to AI. It's the first real vision-only robot to be in my home and I bet it won't be the last from this company. Real honor that they invited me over with my Insta360 camera (another company launched in my home, just like Matic was last year). In Part II we go into the factory.
14:24

Can United States manufacture robots? Matic Robots says "yes." It makes the best floor cleaning robot, that has won many perfect scores from Wired to many others. We love ours. But my trip there to get a tour from AI pioneer Navneet Dalal Navneet Dalal provided some real insights into how hard it is for a hardware company to make hardware in the United States. And how deeply AI is changing consumer electronics products that are going to be in many more homes soon. In this first part (Part II coming tomorrow) we get a look at how long it took for this company to go through prototypes to a shipping product. In the second part, you'll see the scaling hell that it takes to even ship a few thousand robots and the kinds of problems that scaling up a factory brings. Matic is one of my favorite small Silicon Valley companies. It has found what we call "product market fit." I just came back from CES where I saw many of its competitors, and the Matic wins because of not just the product thinking of Mehul and Navneet Dalal but because of their AI leadership. In a way their robot took many lessons from Tesla, from where to put the batteries to its bet on computer vision, which Navneet has been a pioneer in for years, working quietly behind the scenes. It is about to move into a new location that will allow it to grow to meet the demand that now is showing up (the boxes in its lobby show that it's outgrowing its current facilities). In terms of AI, it has aspirations of making a humanoid too, but it is taking a far more measured approach to getting there. By starting on the floor it can not just build world models based on real world data (customers are given a choice whether to allow its data to be used that way. Most customers choose to keep their data on the robot only, for privacy reasons, but if you opt in you can help them improve their models). They are using that data to understand homes. Navneet told me they hit very unusual situations in people's homes already that they couldn't really predict in simulators, like full-wall mirrors that confuse computer vision systems, or pools and water features in people's homes. Having real customers brings a ton of customer feedback about how to further improve the robot, and, as Navneet demonstrates in the second video, forces them to build a manufacturing muscle memory. Getting teams to work together, figuring out how to solve supply chain problems, from Trump's tarriffs, to a new one that showed up over the past couple of weeks. A supplier for its bags (one of the cheaper parts that goes into the robot) changed the glue it used, which caused robots to fail quality tests and the manufacturing line to stop. Reminds me a lot of the hell Elon Musk faced in its Fremont factory when Tesla was first starting to manufacture its Model 3, which almost bankrupted the company. Off the record Mehul and Navneet 🇮🇳 showed me some of the prototypes and plans for its next products that will show up over the next few years. Certainly not as sexy as Tesla, Figure, 1x_tech, and all the Chinese manufacturers are showing off already, but far better thought out for the typical Western home and AI plays a huge role in its future. It is the product that speaks for itself. It's amazing, and is about to get better this year due to AI. It's the first real vision-only robot to be in my home and I bet it won't be the last from this company. Real honor that they invited me over with my Insta360 camera (another company launched in my home, just like Matic was last year). In Part II we go into the factory.

Robert Scoble

69,229 Aufrufe • vor 8 Monaten