Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

For roboticists, one challenge towers above the others: there isn’t enough data. To accelerate the deployment of intelligent robots in the real world, MIT CSAIL’s "LucidSim" uses genAI & physics engines to create diverse & realistic virtual training grounds for robots. W/o any real-world data, their robot achieved expert-level... show more

MIT CSAIL

348,044 subscribers

82,915 views • 1 year ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

9 Comments

MIT CSAIL1 year ago

ChatGPT comes up w/diverse descriptions of the environment, which are then transformed into realistic pictures by a ControlNet. Guidance from the physics engine ensures that images reflect correct collision geometry and real-world physics.

MIT CSAIL1 year ago

To make short, 140 millisecond videos that serve as visual "experiences" for the robot, the scientists hacked together a trick called "Dreams In Motion (DIM)" using a mix of image magic. This trick made LucidSim 7x times faster by moving pixels of a single generated image according to the head movement of the robot, & piece together a short, multi-frame video.

MIT CSAIL1 year ago

LucidSim used AI-generated images to train a robot dog to do parkour — w/o real-world data.

MIT CSAIL1 year ago

Having the dog learn from its own actions is essential. The team put LucidSim against the alternative, where an expert teacher provides substantially more training data. Robots learn from expert data only succeeded 15% of the time — and even quadrupling the amount of expert training data barely moved the needle. When robots collected their own training data through LucidSim, the story changed dramatically: Just doubling the dataset size catapulted success rates to 88%.

MIT CSAIL1 year ago

LucidSim outperforms domain randomization, a go-to method from 2017 that produces diverse data, but lacks realism. LucidSim addresses both diversity & realism problems, helping a robot recognize and navigate obstacles in real environments.

MIT CSAIL1 year ago

LucidSim is paving the way toward a new generation of intelligent machines that elevate & amplify human effort. The researchers envision these machines could safely and autonomously learn how to navigate our complex world w/o ever setting foot in it. The team’s immediate plan is to scale up to a human-sized robot and teach it to manipulate objects on the move, using purely synthesized images.

MIT CSAIL1 year ago

Authors: Alan Yu (@alany1_), Ge Yang (@EpisodeYang), Ran Choi, Yajvan Ravan, John Leonard (@jleonardmit), & Phillip Isola (@phillip_isola) Paper: Full video: Website:

Freedom_Aint_Free1 year ago

There's no enough data ? You mean current algorithms are dumb as a brick, children don't need to look at 1 million pictures of cats to learn what a cat is.

wes george1 year ago

The idea of walking your beta robot...seems like there is a song in that....

Related Videos

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇

Tony Tao

97,402 views • 1 year ago

the micro1 robotics lab: real world data for intelligent models that co-exist in the physical world. we’re in-the-wild across 75 countries in 6,000+ unique environments collecting data. diverse movements, objects, and settings. the future of AI is as human as you can imagine. join us to start training robots today (link in comments).

the micro1 robotics lab: real world data for intelligent models that co-exist in the physical world. we’re in-the-wild across 75 countries in 6,000+ unique environments collecting data. diverse movements, objects, and settings. the future of AI is as human as you can imagine. join us to start training robots today (link in comments).

Ali Ansari

208,233 views • 5 months ago

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes even*cheaper* with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes evencheaper with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

Abhishek Gupta

13,358 views • 1 year ago

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

NVIDIA Robotics

13,479 views • 6 months ago

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”

Elon just dropped a MAJOR nugget on how Tesla is going to be training Optimus to do real world tasks. They are building an Optimus Academy, which is a large scale, dedicated real-world training facility to accelerate the development of Optimus. The Academy will deploy thousands of Optimus units, potentially 10,000 to 30,000 robots, in a controlled realistic environment where they perform self-play, experiment with tasks, iterate on behaviors, and continuously generate training data through trial and error. The Tesla bots will also run millions of simulations in Tesla’s high-fidelity physics-accurate engine, allowing Optimus to close the “sim-to-real gap” by using these real-world observations to refine and validate the simulations! “You’re actually highlighting an important limitation and difference from cars. We’ll soon have 10 million cars on the road. It’s hard to duplicate that massive training flywheel. For the robot, what we’re going to need to do is build a lot of robots and put them in kind of an Optimus Academy so they can do self-play in reality. We’re actually building that out. We can have at least 10,000 Optimus robots, maybe 20-30,000, that are doing self-play and testing different tasks. Tesla has quite a good reality generator, a physics-accurate reality generator, that we made for the cars. We’ll do the same thing for the robots. We actually have done that for the robots. So you have a few tens of thousands of humanoid robots doing different tasks. You can do millions of simulated robots in the simulated world. You use the tens of thousands of robots in the real world to close the simulation to reality gap. Close the sim-to-real gap.”

Teslaconomics

42,563 views • 6 months ago

Today Mecka is announcing $60M in funding to become the data and deployment layer for physical AI This raise will allow us to scale our data infrastructure, invest into new verticals, and deploy robots into the real world

Today Mecka is announcing $60M in funding to become the data and deployment layer for physical AI This raise will allow us to scale our data infrastructure, invest into new verticals, and deploy robots into the real world

Josh

328,775 views • 2 months ago

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

Ilir Aliu - eu/acc

10,509 views • 1 year ago

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

Zhenyu Jiang

44,440 views • 1 year ago

Meet Any2Track, a new two-stage RL framework designed to address a core challenge for humanoid robots: reliably tracking diverse, dynamic motions in the real world. The framework consists of two key components: AnyTracker, a general motion tracker, and AnyAdapter, a module that allows the robot to adapt to real-world disturbances like uneven terrain or external forces. This approach gives robots the stability needed for practical use. The system has been successfully deployed on a Unitree G1 humanoid robot with zero-shot sim2real transfer, performing exceptionally well in various motion tracking tasks under multiple real-world disturbances. Project: Codebase:

Meet Any2Track, a new two-stage RL framework designed to address a core challenge for humanoid robots: reliably tracking diverse, dynamic motions in the real world. The framework consists of two key components: AnyTracker, a general motion tracker, and AnyAdapter, a module that allows the robot to adapt to real-world disturbances like uneven terrain or external forces. This approach gives robots the stability needed for practical use. The system has been successfully deployed on a Unitree G1 humanoid robot with zero-shot sim2real transfer, performing exceptionally well in various motion tracking tasks under multiple real-world disturbances. Project: Codebase:

RoboHub🤖

61,378 views • 11 months ago

Training fully autonomous robots takes more data than the real world can easily provide. Simulation lets us train, test, and deploy autonomous software without being limited by robots, space, or cost. Here’s OM1 running in simulation side-by-side to the real world.

Training fully autonomous robots takes more data than the real world can easily provide. Simulation lets us train, test, and deploy autonomous software without being limited by robots, space, or cost. Here’s OM1 running in simulation side-by-side to the real world.

OpenMind

120,963 views • 8 months ago

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,334 views • 2 months ago

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk

ELON: TESLA’S ROBOTS ARE MOVING FROM SIMULATIONS TO REALITY “For the robot, we’re going to need to build a lot of them and put them in an Optimus Academy so they can do self-play in reality and test different tasks. Tesla has quite a good, physics accurate, reality-generator, and we’re doing that for the robots. So you can do millions of robots in a simulated world, and then 10k in the real world to close the simulation to reality gap. Grok would orchestrate the behavior of the Optimus robots.” Source: Cheeky Pint Podcast, Elon Musk

Mario Nawfal

19,289 views • 6 months ago

Everyone’s talking about robots in factories. Learning physics. Learning touch. Learning to perceive the world. Physical AI’s first lesson is labor. But there’s a harder problem nobody’s talking about. Robots aren’t just going to factories. Museums. Malls. Hospitals. Homes. They have to learn to be with people. That’s exactly what Engineered Arts has been doing for twenty years. While most teams are still training in labs, Ameca is already out there — looking people in the eye, talking back, adapting in real time. Not scripted. Not pre-programmed. Every interaction is training data for understanding humans. Fei-Fei Li said the biggest bottleneck for humanoid robots is data. And the scarcest data? Real human-robot interaction. There’s no shortcut. Just scenes, time, and patience. The trend is right. Lucky we started twenty years ago.

Everyone’s talking about robots in factories. Learning physics. Learning touch. Learning to perceive the world. Physical AI’s first lesson is labor. But there’s a harder problem nobody’s talking about. Robots aren’t just going to factories. Museums. Malls. Hospitals. Homes. They have to learn to be with people. That’s exactly what Engineered Arts has been doing for twenty years. While most teams are still training in labs, Ameca is already out there — looking people in the eye, talking back, adapting in real time. Not scripted. Not pre-programmed. Every interaction is training data for understanding humans. Fei-Fei Li said the biggest bottleneck for humanoid robots is data. And the scarcest data? Real human-robot interaction. There’s no shortcut. Just scenes, time, and patience. The trend is right. Lucky we started twenty years ago.

Engineered Arts

15,549 views • 13 days ago

Axis is officially LIVE on Base. 🔵 Axis is scaling Physical AI for the real world, contributed by everyone. You can control robots in a virtual world, generate training data at scale, and help build the brain behind tomorrow's robots. All from browser. No hardware needed. Start building robotics intelligence today:

Axis is officially LIVE on Base. 🔵 Axis is scaling Physical AI for the real world, contributed by everyone. You can control robots in a virtual world, generate training data at scale, and help build the brain behind tomorrow's robots. All from browser. No hardware needed. Start building robotics intelligence today:

Axis Robotics

324,174 views • 4 months ago

What if robots could improve themselves by learning from their own failures in the real-world? Introducing 𝗣𝗟𝗗 (𝗣𝗿𝗼𝗯𝗲, 𝗟𝗲𝗮𝗿𝗻, 𝗗𝗶𝘀𝘁𝗶𝗹𝗹) — a recipe that enables Vision-Language-Action (VLA) models to self-improve for high-precision manipulation tasks. PLD couples real-world residual reinforcement learning with standard supervised fine-tuning — letting robots discover, recover, and distill their own data flywheel. Quick 🧵

What if robots could improve themselves by learning from their own failures in the real-world? Introducing 𝗣𝗟𝗗 (𝗣𝗿𝗼𝗯𝗲, 𝗟𝗲𝗮𝗿𝗻, 𝗗𝗶𝘀𝘁𝗶𝗹𝗹) — a recipe that enables Vision-Language-Action (VLA) models to self-improve for high-precision manipulation tasks. PLD couples real-world residual reinforcement learning with standard supervised fine-tuning — letting robots discover, recover, and distill their own data flywheel. Quick 🧵

Wenli Xiao

185,353 views • 9 months ago

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Abhishek Gupta

34,563 views • 1 month ago

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Turning video into humanoid robot motion! 🤳🏼 Training humanoid robots needs huge amounts of motion data, but real-world capture doesn’t scale. Mocap is expensive, dangerous edge cases are rare, and you can’t ask humans to repeatedly fall or crash. Video2Robot tackles this by converting videos into physics-grounded humanoid simulations. Motion is generated to respect balance, inertia, ground contact, and joint limits, then directly retargeted to robot simulators. One prompt can generate a full humanoid motion sequence, including multi-agent interactions and failure cases like falls or collisions, scenarios that are hard or impossible to capture safely in the real world. The pipeline is model-agnostic and works with existing video generators, making it a practical way to scale data for robots. If robots are going to operate in the real world, they need to be trained on the failures too, not just the perfect demos. Here's the GitHub: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

59,835 views • 7 months ago

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

Pascal Bornet

186,551 views • 4 months ago

Chelsea Finn (Chelsea Finn) on building general-purpose robotics, and bringing intelligence into the physical world. At AI Startup School in San Francisco. 00:00 - General Purpose Robots 00:11 - Challenges in Robotics Applications 00:57 - Physical Intelligence: A New Approach 01:47 - Learning from Language Models 02:08 - Data Sources for Training Robots 03:32 - Training with Real-World Data 04:39 - Initial Successes and Challenges 09:10 - Breakthrough in Robot Training 11:03 - Improving Performance 15:43 - Expanding Capabilities 17:34 - Robots in Unseen Environments 25:54 - Handling Open-Ended Prompts 29:36 - Evaluating Robot Performance 30:03 - Future Directions and Challenges 31:27 - Audience Q&A

Chelsea Finn (Chelsea Finn) on building general-purpose robotics, and bringing intelligence into the physical world. At AI Startup School in San Francisco. 00:00 - General Purpose Robots 00:11 - Challenges in Robotics Applications 00:57 - Physical Intelligence: A New Approach 01:47 - Learning from Language Models 02:08 - Data Sources for Training Robots 03:32 - Training with Real-World Data 04:39 - Initial Successes and Challenges 09:10 - Breakthrough in Robot Training 11:03 - Improving Performance 15:43 - Expanding Capabilities 17:34 - Robots in Unseen Environments 25:54 - Handling Open-Ended Prompts 29:36 - Evaluating Robot Performance 30:03 - Future Directions and Challenges 31:27 - Audience Q&A

Y Combinator

129,834 views • 1 year ago