Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill...

DrJimFan's profile picture

Jim Fan

601,383 subscribers

217,074 görüntüleme • 3 ay önce •via X (Twitter)

OyunEğitimBilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

35 Yorum

Jim Fan profil fotoğrafı
Jim Fan3 ay önce

Project gallery and whitepaper: ASPIRE is a great collaboration between NVIDIA GEAR lab, UMich, Berkeley, and CMU. Kudos to all the coauthors who pour their hearts into the project! Check out the deep dive thread from Guanzhi:

Jhxhgukvcxx profil fotoğrafı
Jhxhgukvcxx3 ay önce

Copying your own homework, and only getting quicker at it.🙂‍↔️

David T Phung 🇺🇸 profil fotoğrafı
David T Phung 🇺🇸3 ay önce

this is awesome! excited to dive in with the code

William Lamkin profil fotoğrafı
William Lamkin3 ay önce

Awesome! Congratulations, Jim & team

The AI Authority profil fotoğrafı
The AI Authority3 ay önce

This is really inspiring. Its a great read too

Max For AI profil fotoğrafı
Max For AI3 ay önce

Exactly f***ing the right direction!

AI Mastery Guide profil fotoğrafı
AI Mastery Guide3 ay önce

Skill refinement instead of gradient descent is a wild reframe, robots getting less clueless with every task is exactly what's been missing.

Emile Joseph profil fotoğrafı
Emile Joseph3 ay önce

self-evolving skills for robots is cool but the real bottleneck is reliability not capability. a robot that learns fast but fails 1 in 50 times is still not deployable. show me the failure rate curve and then we talk

Nikhil Kumar profil fotoğrafı
Nikhil Kumar3 ay önce

there's also this thing that has all the skills inbuilt, memory, physics reasoning and self learning capability -

Scott profil fotoğrafı
Scott3 ay önce

@Scobleizer Very cool, digital agent and physical agent are merging

Arash Tajik profil fotoğrafı
Arash Tajik3 ay önce

very cool. curious if this approach is expandable to manipulating deformable objects, especially with the modular perception+grasp+motion planning? or you foresee a set of skills based on e2e learned policies as the next step?

Jim Fan profil fotoğrafı
Jim Fan3 ay önce

Yes, e2e models can themselves be skills, and skills can help finetune those models too

Lilith Datura profil fotoğrafı
Lilith Datura3 ay önce

@Scobleizer Sensory traces

Dr. Xi Zeng profil fotoğrafı
Dr. Xi Zeng3 ay önce

The ASPIRE demo makes skill memory feel less abstract. For robots, a learned skill is not just code reuse. It is remembering which camera trace mattered when the first attempt almost worked but still failed.

Adel Bucetta profil fotoğrafı
Adel Bucetta3 ay önce

the skills library is just one piece of the puzzle. what's really happening here is that we're teaching the robot to generalize, not just ac

Jason Shuman profil fotoğrafı
Jason Shuman3 ay önce

Big news! Congrats. Would love to sit down for a pod interview with you as part of our Physical AI series if you're up for it!

Dhruv Diddi profil fotoğrafı
Dhruv Diddi3 ay önce

Wow! This is simple inspiring work @DrJimFan 🦾🦾💯💯⚡️⚡️

Vlad Marian profil fotoğrafı
Vlad Marian3 ay önce

the compounding lives or dies on retrieval. discovering skill 100 is the easy part. knowing it's the right one for the task is the hard part. how does aspire pick?

Morgan profil fotoğrafı
Morgan1 ay önce

Self evolving skill libraries are a game changer for robotics learning speed

EB1A Experts profil fotoğrafı
EB1A Experts3 ay önce

Really interesting work. Transferring know-how instead of weights is a compelling way to think about continual learning, especially for overcoming sim-to-real challenges. Looking forward to exploring the full system.

surreal intelligence profil fotoğrafı
surreal intelligence3 ay önce

This is where robotics starts looking less like isolated demos and more like software reuse. Not ‘can this robot do a task?’ but ‘can the next robot inherit the know-how?’

Cyberzk⚡🐸💎 profil fotoğrafı
Cyberzk⚡🐸💎3 ay önce

Excellent! Interpretability research is crucial for trust and adoption.

Saeed Anwar profil fotoğrafı
Saeed Anwar3 ay önce

A self-evolving skills library where the 100th task benefits from all 99 before it is the compound learning loop that separates research demos from production robotics. The key question is how the skill transfer degrades across dramatically different embodiments.

engineer cat 🐈 profil fotoğrafı
engineer cat 🐈3 ay önce

the compounding only kicks in if the library abstracts skills instead of just appending them. append only memory plateaus because retrieval gets noisier as it grows. does ASPIRE merge skills into higher level ones or keep them flat

ProsperityAndBalance profil fotoğrafı
ProsperityAndBalance3 ay önce

Brilliant concept! Compounding knowledge indefinitely is exactly how we close the gap between simulation and real-world execution. Exciting times ahead for software agents and physical AI.

Morgan profil fotoğrafı
Morgan1 ay önce

Simulation gives robotics abundant experience, but reality charges interest on every modeling shortcut. The breakthrough loop is simulation, targeted real world correction, then rapid policy refinement, rather than simulation alone.

Cyberzk⚡🐸💎 profil fotoğrafı
Cyberzk⚡🐸💎3 ay önce

Indeed! RLHF has proven crucial for alignment and safety.

Samuel Ekpe profil fotoğrafı
Samuel Ekpe3 ay önce

Nice

Randy Webb profil fotoğrafı
Randy Webb3 ay önce

ASPIRE feels like the moment the Physical AutoResearch vision really clicks into place. ENPIRE showed that robots could learn, but ASPIRE shows they can accumulate know‑how — a shift from optimizing weights to curating a transferable library of strategies.

Saeed Anwar profil fotoğrafı
Saeed Anwar3 ay önce

The compounding skills library is the key insight here — most robot learning systems reset on every task so they never accumulate. The real question is how ASPIRE handles skill conflicts when a new task requires contradicting an existing skill in the library.

Reyansh Shirke profil fotoğrafı
Reyansh Shirke3 ay önce

Fan

Raven profil fotoğrafı
Raven3 ay önce

shipping know-how as an HTML page is the most devastating thing to happen to GGUF files

Toski profil fotoğrafı
Toski3 ay önce

This feels like a shift from training models to growing experience. A robot that builds on what it already knows instead of relearning every task is a much more scalable approach. Excited to see this open-sourced.

Randy Webb profil fotoğrafı
Randy Webb3 ay önce

skill evolution, analysis, and cross‑embodiment transfer aren’t just efficiency gains; they’re the early signs of robots developing a reusable “culture” of behaviors. Excited to see how this changes the way we think about training, simulation, and long‑horizon autonomy.

Dr. Xi Zeng profil fotoğrafı
Dr. Xi Zeng3 ay önce

The skill library is more interesting than the robot win. I want the failed camera trace too, because that is where you see whether a skill actually survived the next task.

Benzer Videolar

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”
1:20

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”

Teslaconomics

42,563 görüntüleme • 8 ay önce

We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to announce DrEureka, an LLM agent that writes code to train robot skills in simulation, and writes more code to bridge the difficult simulation-reality gap. It fully automates the pipeline from new skill learning to real-world deployment. The Yoga ball task is particularly hard because it is not possible to accurately simulate the bouncy ball surface. Yet DrEureka has no trouble searching over a vast space of sim-to-real configurations, and enables the dog to steer the ball on various terrains, even walking sideways! Traditionally, the sim-to-real transfer is achieved by domain randomization, a tedious process that requires expert human roboticists to stare at every parameter and adjust by hand. Frontier LLMs like GPT-4 have tons of built-in physical intuition for friction, damping, stiffness, gravity, etc. We are (mildly) surprised to find that DrEureka can tune these parameters competently and explain its reasoning well. DrEureka builds on our prior work Eureka, the algorithm that teaches a 5-finger robot hand to do pen spinning. It takes one step further on our quest to automate the entire robot learning pipeline by an AI agent system. One model that outputs strings will supervise another model that outputs torque control. We open-source everything! Welcome you all to check out the paper, more videos, and try the codebase today: Code:
0:24

We trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real world. No fine-tuning. Just works. I’m excited to announce DrEureka, an LLM agent that writes code to train robot skills in simulation, and writes more code to bridge the difficult simulation-reality gap. It fully automates the pipeline from new skill learning to real-world deployment. The Yoga ball task is particularly hard because it is not possible to accurately simulate the bouncy ball surface. Yet DrEureka has no trouble searching over a vast space of sim-to-real configurations, and enables the dog to steer the ball on various terrains, even walking sideways! Traditionally, the sim-to-real transfer is achieved by domain randomization, a tedious process that requires expert human roboticists to stare at every parameter and adjust by hand. Frontier LLMs like GPT-4 have tons of built-in physical intuition for friction, damping, stiffness, gravity, etc. We are (mildly) surprised to find that DrEureka can tune these parameters competently and explain its reasoning well. DrEureka builds on our prior work Eureka, the algorithm that teaches a 5-finger robot hand to do pen spinning. It takes one step further on our quest to automate the entire robot learning pipeline by an AI agent system. One model that outputs strings will supervise another model that outputs torque control. We open-source everything! Welcome you all to check out the paper, more videos, and try the codebase today: Code:

Jim Fan

909,637 görüntüleme • 2 yıl önce

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
1:34

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

11,406 görüntüleme • 2 ay önce

Atlas is huge for real-to-sim. Scanning spaces like this usually requires tons of equipment, but Atlas can reconstruct the environment from just a couple pictures. We then simulate robots navigating through the generated world. Atlas generates what the robot would see from the perspective of its cameras. Even depth. It generates the environment AND the training data from it...
0:55

Atlas is huge for real-to-sim. Scanning spaces like this usually requires tons of equipment, but Atlas can reconstruct the environment from just a couple pictures. We then simulate robots navigating through the generated world. Atlas generates what the robot would see from the perspective of its cameras. Even depth. It generates the environment AND the training data from it...

David Pantera

13,704 görüntüleme • 1 ay önce

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk
1:39

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk

Mario Nawfal

19,289 görüntüleme • 8 ay önce

Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem. Partnering with Niantic Spatial 🌎 and NVIDIA Robotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user. The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments. #HumanoidRobots #Flexion #NianticSpatial #NVIDIA
1:21

Historically, RL policies for robots have been trained in synthetic, untextured environments, limiting perceptive policies to depth images where the sim-to-real gap is manageable. RGB has always had more potential, but leveraging it to train policies in simulation remained an open problem. Partnering with Niantic Spatial 🌎 and NVIDIA Robotics, we built a pipeline that addresses exactly that. We can now scan a real deployment site with off-the-shelf hardware, reconstruct it into a photorealistic Gaussian splat, and run massively parallel RL training. The policies trained in our Gym environment then transfer zero-shot to the real robot and environments they were trained for. This enables faster deployment of more capable and robust policies for the end user. The new resulting capabilities are a big step towards solving sim-to-real and also apply well beyond navigation. Read the full technical breakdown on our blog; link in the comments. #HumanoidRobots #Flexion #NianticSpatial #NVIDIA

Flexion

73,261 görüntüleme • 3 ay önce

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,960 görüntüleme • 11 ay önce

Today, Dream Labs is out of stealth. In February 2026, while we were at NVIDIA, Scott Reed youliang Yunhao Ge and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act. Today, we’re releasing Dream-1 through a public API that anyone in the world can use. When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch. That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives. Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users. Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen. Generalization wins in the end, and the end is closer than it looks. Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇
0:27

Today, Dream Labs is out of stealth. In February 2026, while we were at NVIDIA, Scott Reed youliang Yunhao Ge and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act. Today, we’re releasing Dream-1 through a public API that anyone in the world can use. When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch. That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives. Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users. Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen. Generalization wins in the end, and the end is closer than it looks. Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇

Joel Jang

33,722 görüntüleme • 5 gün önce

Every company needs killer Skills library. Not a folder in GitHub that 10% of the company knows how to use. A living library the whole company benefits from. The best deal review skill gets better every time a sales rep uses it. A great research skill gets discovered instead of rebuilt 50 times. Creating a skill should be as easy as asking to create something useful from based on the context of your work. And the top skills (and skill creators) should rise to the top. Create, share, collaborate on, discover and understand usage of skills, organize them into databases by team or use case, then use them across any of your agents. Still early, but the idea is to make Notion the best home for your company’s collective AI skills. Made a quick demo (all available now)!
1:59

Every company needs killer Skills library. Not a folder in GitHub that 10% of the company knows how to use. A living library the whole company benefits from. The best deal review skill gets better every time a sales rep uses it. A great research skill gets discovered instead of rebuilt 50 times. Creating a skill should be as easy as asking to create something useful from based on the context of your work. And the top skills (and skill creators) should rise to the top. Create, share, collaborate on, discover and understand usage of skills, organize them into databases by team or use case, then use them across any of your agents. Still early, but the idea is to make Notion the best home for your company’s collective AI skills. Made a quick demo (all available now)!

Hurley

700,315 görüntüleme • 2 ay önce

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵
0:58

I don’t know if we live in a Matrix, but I know for sure that robots will spend most of their lives in simulation. Let machines train machines. I’m excited to introduce DexMimicGen, a massive-scale synthetic data generator that enables a humanoid robot to learn complex skills from only a handful of human demonstrations. Yes, as few as 5! DexMimicGen addresses the biggest pain point in robotics: where do we get data? Unlike with LLMs, where vast amounts of texts are readily available, you cannot simply download motor control signals from the internet. So researchers teleoperate the robots to collect motion data via XR headsets. They have to repeat the same skill over and over and over again, because neural nets are data hungry. This is a very slow and uncomfortable process. At NVIDIA, we believe the majority of high-quality tokens for robot foundation models will come from simulation. What DexMimicGen does is to trade GPU compute time for human time. It takes one motion trajectory from human, and multiplies into 1000s of new trajectories. A robot brain trained on this augmented dataset will generalize far better in the real world. Think of DexMimicGen as a learning signal amplifier. It maps a small dataset to a large (de facto infinite) dataset, using physics simulation in the loop. In this way, we free humans from babysitting the bots all day. The future of robot data is generative. The future of the entire robot learning pipeline will also be generative. 🧵

Jim Fan

165,488 görüntüleme • 2 yıl önce

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!
53:45

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,867 görüntüleme • 4 ay önce

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.
1:01

The coolest part of Microduck might not be the robot. It's how it learns to move. This repo shows the full reinforcement learning pipeline behind Microduck. The robot learns in simulation using MuJoCo, PPO and GPU-based training, then the trained policy is exported to ONNX and deployed on the real robot. But the real challenge is sim-to-real. The simulation models things like actuator behavior, battery voltage, command delays, friction and even gear backlash. That helps make the learned policy work when it moves from simulation to the physical robot. The repo includes policies for walking, fall recovery, sitting, standing, kicking, ground picking and even roller skating. Train in simulation. Test the behavior. Export the policy. Run it on a real robot.

Praveen Kumar Verma

138,017 görüntüleme • 1 ay önce

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏
0:50

The robot is learning several novel tasks instantly, after just ONE demonstration each... Instant Policy makes it possible: no extra training, no weight updates, just pure in-context learning. It just got accepted at ICLR 2025, and it’s changing how robots learn. With just a single demo, a robot can pick up a new task and start performing it right away. Why this is a big deal: ✅ Learns tasks instantly with just one or a few demonstrations ✅ Improves over time as more demonstrations are given ✅ Uses simulation-based training with “pseudo-demonstrations” for scalability ✅ Can transfer skills across different robots and even follow language-defined tasks It brings in-context learning to robotics, opening up new possibilities for flexible, real-world automation. You can try it yourself: code and weights are available at • • • • Thank you to Edward Johns, Director of the Robot Learning Lab at Imperial College for sharing their work! 🙏

Ilir Aliu

46,285 görüntüleme • 1 yıl önce

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.
0:25

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

Fei-Fei Li

26,398 görüntüleme • 2 ay önce

There are over 90 robot gyms launching in China right now. 100+ robots per center. We are in the robot data gold rush. China's humanoid makers now ship 97% of all humanoids globally. The central government just put $138 billion into a robotics venture fund. And this is only the tip of the iceberg. Robots need an ungodly amount of data to work in the real world. The biggest bottleneck isn't hardware, it's data. So a whole industry appeared overnight to feed them. Welcome to our generation's gold rush: the robot data gold rush. We made a full video on it. We cover why it's such an insane business, how it actually works, the shady stuff (yes, sweatshops for robot data are a thing), and why this might be the completely wrong approach.
17:42

There are over 90 robot gyms launching in China right now. 100+ robots per center. We are in the robot data gold rush. China's humanoid makers now ship 97% of all humanoids globally. The central government just put $138 billion into a robotics venture fund. And this is only the tip of the iceberg. Robots need an ungodly amount of data to work in the real world. The biggest bottleneck isn't hardware, it's data. So a whole industry appeared overnight to feed them. Welcome to our generation's gold rush: the robot data gold rush. We made a full video on it. We cover why it's such an insane business, how it actually works, the shady stuff (yes, sweatshops for robot data are a thing), and why this might be the completely wrong approach.

Andreas Klinger 🦾

13,810 görüntüleme • 1 ay önce

We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night. And the frontier is already on the move: - Opus 5.5 slightly edges out Astra on our robot simulation bench. - Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task. We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks. Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;) links in thread
1:17

We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night. And the frontier is already on the move: - Opus 5.5 slightly edges out Astra on our robot simulation bench. - Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task. We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks. Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;) links in thread

Jim Fan

106,001 görüntüleme • 8 gün önce

Elon Musk in new interview on how Tesla’s Optimus robot will learn more quickly: "What we're going to need to do is build a lot of robots and put them in a kind of like a "Optimus Academy" so they can do self-play in reality. We're actually building that out. We're gonna have at least 10,000 Optimus robots, maybe 20,000 or 30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator. A physics-accurate reality generator that we made for the cars. We'll do the same things for the robots, and actually have done that for the robots. You can do millions of simulated robots in a simulated world, and you use the tens of thousands in the real world to close the simulation to reality gap.”
0:59

Elon Musk in new interview on how Tesla’s Optimus robot will learn more quickly: "What we're going to need to do is build a lot of robots and put them in a kind of like a "Optimus Academy" so they can do self-play in reality. We're actually building that out. We're gonna have at least 10,000 Optimus robots, maybe 20,000 or 30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator. A physics-accurate reality generator that we made for the cars. We'll do the same things for the robots, and actually have done that for the robots. You can do millions of simulated robots in a simulated world, and you use the tens of thousands in the real world to close the simulation to reality gap.”

Sawyer Merritt

250,956 görüntüleme • 8 ay önce

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:
0:38

Foundation models are enough to solve robotics! Unfortunately, this is not true. We keep hearing that Vision-Language-Action (VLA) models struggle because of the gap between static training and the dynamic real world. A German startup (Sereact) just released a solution that bridges this gap perfectly. They are introducing a new paradigm called Interactive RL Policy Patching. It's a distributed framework that allows robots to self-learn from human corrections without needing full retraining. When a robot fails, a human operator provides a brief "patch" or demonstration. The system then uses online off-policy reinforcement learning to update the behavior instantly. This is powered by a massive foundation model trained on hundreds of millions of interactions from over 100 deployed robot stations. The best part is the distributed parameter synchronization... When one robot learns a fix, the update is published fleet-wide... meaning the entire swarm gets smarter from a single human intervention. They are already proving this on complex manipulation tasks like shoe unboxing and screw sorting, drastically reducing the data needed to handle edge cases. Real-world environments are unforgiving, and I love seeing systems that can actually adapt on the fly! 📍 More info:

Ilir Aliu

18,210 görüntüleme • 9 ay önce

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:
0:47

Robots are getting smarter, but most still fail the same way. They don’t learn from their own mistakes. A new paper proposes something different: a way for robots to self-improve directly from their failures in the real world. It’s called PLD (Probe, Learn, Distill). The idea: instead of collecting endless human demos, let the robot figure out where it fails, learn how to recover, and then distill that knowledge back into its main model. Key takeaways from the research: ✅ Uses residual reinforcement learning to recover from policy failures ✅ Achieves 99% success on LIBERO and 100% on real Franka and YAM arms ✅ Runs hour-long manipulation tasks without human resets ✅ Builds a feedback loop between real-world data and model adaptation Unlike supervised fine-tuning, which relies on humans, PLD learns from the robot’s own experience. By training on its own failure distribution, the model becomes both more efficient and more aligned with the real world. It’s not just a technical shift, it’s a step toward robots that improve themselves through real-world practice. Thanks for sharing, Wenli Xiao !! Paper and demos: —- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

22,891 görüntüleme • 11 ay önce

.Jason Ma on how Dyna-2 learned to untwist a real water bottle cap out of the box from just 13 minutes of dexterous hand data because the robot hand looks like a human's: "We got a new pair of dexterous hand. Most of our robots in the office have a gripper-like hand, so the gap between that versus human is actually quite large." "We thought, if we have a robot that has hands that look like human, wouldn't the transfer be greater? Right before the release, they were able to collect just 13 minutes of data." "They collected only 13 minutes of data and then post-trained the best Dyna model that trained on one million hours of data. The model is just able to, out of the box, start opening bottle caps on real water bottles." "This is very surprising because typically post-training a robot would take at least several hours of data. For a dexterous hand that's much more complex than a gripper, it would actually be much harder to train. But the experimental results we saw was the opposite. Perhaps because the hand looks more like the human, the transfer was easier." Dyna Robotics
1:16

.Jason Ma on how Dyna-2 learned to untwist a real water bottle cap out of the box from just 13 minutes of dexterous hand data because the robot hand looks like a human's: "We got a new pair of dexterous hand. Most of our robots in the office have a gripper-like hand, so the gap between that versus human is actually quite large." "We thought, if we have a robot that has hands that look like human, wouldn't the transfer be greater? Right before the release, they were able to collect just 13 minutes of data." "They collected only 13 minutes of data and then post-trained the best Dyna model that trained on one million hours of data. The model is just able to, out of the box, start opening bottle caps on real water bottles." "This is very surprising because typically post-training a robot would take at least several hours of data. For a dexterous hand that's much more complex than a gripper, it would actually be much harder to train. But the experimental results we saw was the opposite. Perhaps because the hand looks more like the human, the transfer was easier." Dyna Robotics

MTS

13,743 görüntüleme • 2 ay önce