Loading video...

Video Failed to Load

Go Home

AI for robotics has some monumental challenges, like high dimensionality and reasoning with the physical world. Ashok Elluswamy, AI lead at Tesla, thinks the hardest problem is Evaluation. - Loss isn't a perfect indicator of Policy Neural Network's quality. - Testing in the real world for all the long-tail...

TheHumanoidHub's profile picture

The Humanoid Hub

93,302 subscribers

15,907 views • 4 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐
3:04

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

Yunzhu Li

39,850 views • 7 months ago

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation
2:52

#AGIBOTAIWeek Day4: Introducing GE-Sim 2.0 (Genie Envisioner World Simulator 2.0) , a comprehensive embodied world simulator for robotic manipulation. By integrating future video generation, proprioceptive state estimation, and reward-based policy assessment, GE-Sim 2.0 moves beyond visual simulation toward an embodied world simulator with built-in evaluation capability, laying the foundation for a world-model-centric platform for closed-loop and interactive policy learning. It provides a practical framework for scalable evaluation and training of robotic manipulation policies. Project page: #AGIBOT #AGIBOTAIWeek #GenieEnvisioner #Robotics #visualsimulation

AGIBOT

28,219 views • 2 months ago

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.
0:40

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 views • 3 months ago

NEWS: Waymo has introduced the Waymo World Model, "a frontier generative model built on Google DeepMind’s Genie 3 that sets a new bar for large-scale, hyper-realistic autonomous driving simulation." "By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. This combination of broad world knowledge, fine-grained controllability, and multi-modal realism enhances Waymo’s ability to safely scale our service across more places and new driving environments. In the following sections we showcase the Waymo World Model in action, featuring simulations of the Waymo Driver navigating diverse rare edge-case scenarios." More info below.
1:03

NEWS: Waymo has introduced the Waymo World Model, "a frontier generative model built on Google DeepMind’s Genie 3 that sets a new bar for large-scale, hyper-realistic autonomous driving simulation." "By simulating the “impossible”, we proactively prepare the Waymo Driver for some of the most rare and complex scenarios—from tornadoes to planes landing on freeways—long before it encounters them in the real world. The model’s architecture offers high controllability, allowing our engineers to modify simulations with simple language prompts, driving inputs, and scene layouts. Notably, the Waymo World Model generates high-fidelity, multi-sensor outputs that include both camera and lidar data. This combination of broad world knowledge, fine-grained controllability, and multi-modal realism enhances Waymo’s ability to safely scale our service across more places and new driving environments. In the following sections we showcase the Waymo World Model in action, featuring simulations of the Waymo Driver navigating diverse rare edge-case scenarios." More info below.

Sawyer Merritt

133,082 views • 4 months ago

The next frontier of autonomous driving is unlocked by reasoning models. NVIDIA Alpamayo brings together open AI models with reasoning capabilities, closed-loop simulation tools, and massive real-world driving datasets. Alpamayo 1 is a vision–language–action model that explains its own decisions through explicit reasoning traces, enabling trustworthy, humanlike decision-making. Together with NVIDIA’s Physical AI dataset and AlpaSim simulation, Alpamayo provides the tools and scale required to enable level 4 autonomous vehicles. ▶️ Watch now:
0:45

The next frontier of autonomous driving is unlocked by reasoning models. NVIDIA Alpamayo brings together open AI models with reasoning capabilities, closed-loop simulation tools, and massive real-world driving datasets. Alpamayo 1 is a vision–language–action model that explains its own decisions through explicit reasoning traces, enabling trustworthy, humanlike decision-making. Together with NVIDIA’s Physical AI dataset and AlpaSim simulation, Alpamayo provides the tools and scale required to enable level 4 autonomous vehicles. ▶️ Watch now:

NVIDIA DRIVE

35,324 views • 5 months ago

Self-learning worked in AlphaGo because the world was a clean rulebook, says Demis Hassabis. Real life has no perfect rulebook, so long-term planning needs a strong “world model” to simulate what happens next. Without that simulation, a robot has to guess in the real world—where mistakes can be costly and irreversible. Better world models are the first missing piece, says Demis Hassabis.
1:38

Self-learning worked in AlphaGo because the world was a clean rulebook, says Demis Hassabis. Real life has no perfect rulebook, so long-term planning needs a strong “world model” to simulate what happens next. Without that simulation, a robot has to guess in the real world—where mistakes can be costly and irreversible. Better world models are the first missing piece, says Demis Hassabis.

Wes Roth

15,757 views • 4 months ago

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.
0:29

In the era of 10T-parameter foundation models 😈, can we still build next-generation world representations using 🦥 a single RTX 3090 GPU? We introduce WorldString (a name inspired by string theory), a tiny neural network representation for reconstructing dynamic world instances, ranging from articulated motion and skeletal skinning to soft-body deformation. 🔥 We view object representation as the core primitive of world modeling: the basic executable unit from which perception, interaction, simulation, and generation can be built. With a compact neural basis, WorldString aims to connect dynamic object modeling with neural simulation and foundation models, enabling a scalable path toward richer representations of the physical world.

Xueyan Zou

36,747 views • 9 days ago

It’s easy to watch this video and see a pre-recorded clip, but what you’re really watching is a frame-by-frame, interactive simulation of the world. Odyssey-2 Pro—a general-purpose world model—is prompted and then rolls out a minutes-long, open-ended simulation.
1:41

It’s easy to watch this video and see a pre-recorded clip, but what you’re really watching is a frame-by-frame, interactive simulation of the world. Odyssey-2 Pro—a general-purpose world model—is prompted and then rolls out a minutes-long, open-ended simulation.

Odyssey

32,291 views • 6 months ago

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.
1:52

The "reality gap" is narrowing. Sanctuary AI has demonstrated a sim-to-real breakthrough: "zero-shot" transfer for complex in-hand manipulation. By leveraging high-DOF hydraulic hands, the company successfully reoriented a cube in the physical world using a policy trained entirely in simulation.

Humanoids daily

12,391 views • 2 months ago

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 views • 3 months ago

NVIDIA CEO Jensen Huang on Tesla FSD at CES: “Tesla’s FSD stack is completely world-class. They’ve been working on it for quite some time. It’s world-class. Not only in the number of miles it has, but it’s world-class in the way that it’s designed, the way they do training to data collection and curation, synthetic data generation, all of their simulation technologies. Of course, the latest generation is end-to-end full self-driving, meaning it’s just one large model that is end-to-end trained. Elon’s AV system is, in every way, 100% state-of-the-art. And so I’m really quite impressed by the technology. I have it and I drive it in our house and it works incredibly well.”
1:11

NVIDIA CEO Jensen Huang on Tesla FSD at CES: “Tesla’s FSD stack is completely world-class. They’ve been working on it for quite some time. It’s world-class. Not only in the number of miles it has, but it’s world-class in the way that it’s designed, the way they do training to data collection and curation, synthetic data generation, all of their simulation technologies. Of course, the latest generation is end-to-end full self-driving, meaning it’s just one large model that is end-to-end trained. Elon’s AV system is, in every way, 100% state-of-the-art. And so I’m really quite impressed by the technology. I have it and I drive it in our house and it works incredibly well.”

Nic Cruz Patane

52,826 views • 5 months ago

At #GTC26 we are showing AlpaDreams — generative, interactive closed-loop simulation for autonomous driving. World models generate environments in real time, policies or drivers interact, and the simulator updates the world. If you are at GTC, come by the booth and try it out!
0:49

At #GTC26 we are showing AlpaDreams — generative, interactive closed-loop simulation for autonomous driving. World models generate environments in real time, policies or drivers interact, and the simulator updates the world. If you are at GTC, come by the booth and try it out!

Zian Wang

16,766 views • 3 months ago

Robot trained entirely in simulation! 🤯 Sudo AI just released #sudo R1. This could be the moment the data bottleneck in robotics finally breaks, a major step forward for getting simulation-trained robots to actually work in the real world. #sudo R1 is a manipulation foundation model that reliably picks objects it has never seen: glass, fabric, metal, irregular shapes. All with zero real-world training data. No tele-op. No demos. No manual labeling. Pure simulation. And picking isn't just any skill, it's the foundation. Almost everything we want robots to do starts with being able to reliably pick something up. Data collection is the bottleneck holding the whole field back. Every new task means hours of tele-op and armies of labelers. If simulation alone can actually deliver production-grade picking, that bottleneck starts to disappear. Really exciting to see! If the results hold, the path to general-purpose robots just got a whole lot shorter. 👇 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
0:59

Robot trained entirely in simulation! 🤯 Sudo AI just released #sudo R1. This could be the moment the data bottleneck in robotics finally breaks, a major step forward for getting simulation-trained robots to actually work in the real world. #sudo R1 is a manipulation foundation model that reliably picks objects it has never seen: glass, fabric, metal, irregular shapes. All with zero real-world training data. No tele-op. No demos. No manual labeling. Pure simulation. And picking isn't just any skill, it's the foundation. Almost everything we want robots to do starts with being able to reliably pick something up. Data collection is the bottleneck holding the whole field back. Every new task means hours of tele-op and armies of labelers. If simulation alone can actually deliver production-grade picking, that bottleneck starts to disappear. Really exciting to see! If the results hold, the path to general-purpose robots just got a whole lot shorter. 👇 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

25,567 views • 2 months ago

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.
16:27

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.

abdel

16,873 views • 1 month ago

New Truthstream film: "This is the world that you know. The world as it was at the end of the 20th century. It exists now only as part of a neural interactive simulation that we call 'The Matrix'." "2025: Living in a Dream World" {Excerpt}
12:38

New Truthstream film: "This is the world that you know. The world as it was at the end of the 20th century. It exists now only as part of a neural interactive simulation that we call 'The Matrix'." "2025: Living in a Dream World" {Excerpt}

Truthstream Media

34,182 views • 5 months ago

ATTENTION: The World Health Organization, John Hopkins University and the Bill & Melinda Gates Foundation completed a new pandemic simulation back in October of 2022 in Belgium, Germany. These are the same exact people who ran event 201, which was a pandemic simulation they ran for Covid six months before the real Covid Pandemic hit. According to the simulation this new pandemic will originate in Brazil in 2025 and the virus is called #SEERS (Severe, Epidemic, Enterovirus, Respiratory, Syndrome). Reports in the simulation stated there’d be a billion cases worldwide with 20 million deaths, including 15 million children, with countless millions left with paralysis. SEERS is an enterovirus, like Polio, that can not only effect the nerves in the spinal cord, and cause paralysis, but it can also effect the brain stem, and make people brain dead. According to the simulation we have approx a year and half to get as ready as possible. Plan accordingly. #pandemic #pandemic2025 #Genocide
6:09

ATTENTION: The World Health Organization, John Hopkins University and the Bill & Melinda Gates Foundation completed a new pandemic simulation back in October of 2022 in Belgium, Germany. These are the same exact people who ran event 201, which was a pandemic simulation they ran for Covid six months before the real Covid Pandemic hit. According to the simulation this new pandemic will originate in Brazil in 2025 and the virus is called #SEERS (Severe, Epidemic, Enterovirus, Respiratory, Syndrome). Reports in the simulation stated there’d be a billion cases worldwide with 20 million deaths, including 15 million children, with countless millions left with paralysis. SEERS is an enterovirus, like Polio, that can not only effect the nerves in the spinal cord, and cause paralysis, but it can also effect the brain stem, and make people brain dead. According to the simulation we have approx a year and half to get as ready as possible. Plan accordingly. #pandemic #pandemic2025 #Genocide

Lion of Judah

581,666 views • 3 years ago

2. NVIDIA is building the GPT of humanoid robots. They just launched Isaac GR00T N1.5 - a foundation model for general purpose robotics. Here’s how it works: → A human demos the task once → Cosmos (their physics AI model) generates 1,000s of variations → Omniverse simulates the motions in high fidelity → The robot trains entirely in simulation → Then fine-tunes itself in the real world Robots can now learn general skills across tasks, tools, even body types with just one human demo. AI isn’t just thinking in text anymore. It’s perceiving. Reasoning. Moving. Physical AI is here and it’s training itself.
2:21

2. NVIDIA is building the GPT of humanoid robots. They just launched Isaac GR00T N1.5 - a foundation model for general purpose robotics. Here’s how it works: → A human demos the task once → Cosmos (their physics AI model) generates 1,000s of variations → Omniverse simulates the motions in high fidelity → The robot trains entirely in simulation → Then fine-tunes itself in the real world Robots can now learn general skills across tasks, tools, even body types with just one human demo. AI isn’t just thinking in text anymore. It’s perceiving. Reasoning. Moving. Physical AI is here and it’s training itself.

Shruti

70,359 views • 1 year ago

Why Robots work in Simulation but fail in Reality One of the most frustrating moments in robotics: Everything works perfectly in simulation. Then you deploy it on a real robot and suddenly: The grasp misses The arm shakes The robot drifts Contact becomes unstable The motion looks correct, but the task still fails How do you solve the sim to real problem? At first, it sounds simple. Just move the code from simulation onto hardware. But the gap between simulation and reality is much larger than most people think. Simulation environments are extremely clean. The table is flat. Object geometry is accurate. Friction is predefined. Sensors are stable. Robot joints behave exactly as expected. But the real world is messy. Lighting changes. Depth sensors drift. Objects reflect light differently. Motors have delay. Joints have backlash. Contact forces behave unpredictably. And robotics is a chain reaction. A small perception error becomes a planning error. The planning error becomes a control error. The control error becomes an execution error. Eventually, the robot misses the grasp by a few centimeters and the entire task fails. And The hardest part is usually contact. Humans think tasks like: grasping a cup, opening a door, inserting an object, pushing a box are trivial. For robots, these are extremely difficult because contact is not clean physics. A tiny shift in friction, force, or surface geometry can completely change the result. In simulation, objects are usually “well behaved.” In reality: objects slip contact points shift surfaces deform collisions happen unexpectedly This is why many robotic tasks fail not because the policy is fundamentally wrong, but because reality itself introduces uncertainty. Sensors are also less reliable than people think. The robot’s perception already contains error: camera noise unstable depth estimation occlusion pose estimation drift changing lighting conditions Sometimes the model itself is fine, but the input is already slightly wrong. By the time the error propagates to the end effector, the grasp fails. The robot hardware itself is also imperfect. Motors have latency. Controllers have frequency limits. Actuators have error. Different loads change behavior. In simulation, the robot follows commands perfectly. In reality, it may move slightly slower, slightly off target, or slightly unstable. Those tiny differences are fatal in robotics because robots physically interact with the world. Sim2Real being difficult does not mean simulation is useless. Simulation is still incredibly valuable: they are cheap, safe, scalable and reproducible. A better way to think about simulation is: Simulation is the training ground, not the final battlefield. Modern Sim2Real methods usually combine multiple approaches: making simulation more realistic, adding domain randomization, randomizing lighting, friction, object positions, and sensor noise, fine-tuning with real-world data. The goal is not to make the robot adapt to one perfect virtual world. The goal is to make the robot robust enough to survive an imperfect real one. The most important lesson in robotics is: Success in simulation is only the first step. The real test begins when the robot touches the real world. Video Credit: Kevin Zakka
0:20

Why Robots work in Simulation but fail in Reality One of the most frustrating moments in robotics: Everything works perfectly in simulation. Then you deploy it on a real robot and suddenly: The grasp misses The arm shakes The robot drifts Contact becomes unstable The motion looks correct, but the task still fails How do you solve the sim to real problem? At first, it sounds simple. Just move the code from simulation onto hardware. But the gap between simulation and reality is much larger than most people think. Simulation environments are extremely clean. The table is flat. Object geometry is accurate. Friction is predefined. Sensors are stable. Robot joints behave exactly as expected. But the real world is messy. Lighting changes. Depth sensors drift. Objects reflect light differently. Motors have delay. Joints have backlash. Contact forces behave unpredictably. And robotics is a chain reaction. A small perception error becomes a planning error. The planning error becomes a control error. The control error becomes an execution error. Eventually, the robot misses the grasp by a few centimeters and the entire task fails. And The hardest part is usually contact. Humans think tasks like: grasping a cup, opening a door, inserting an object, pushing a box are trivial. For robots, these are extremely difficult because contact is not clean physics. A tiny shift in friction, force, or surface geometry can completely change the result. In simulation, objects are usually “well behaved.” In reality: objects slip contact points shift surfaces deform collisions happen unexpectedly This is why many robotic tasks fail not because the policy is fundamentally wrong, but because reality itself introduces uncertainty. Sensors are also less reliable than people think. The robot’s perception already contains error: camera noise unstable depth estimation occlusion pose estimation drift changing lighting conditions Sometimes the model itself is fine, but the input is already slightly wrong. By the time the error propagates to the end effector, the grasp fails. The robot hardware itself is also imperfect. Motors have latency. Controllers have frequency limits. Actuators have error. Different loads change behavior. In simulation, the robot follows commands perfectly. In reality, it may move slightly slower, slightly off target, or slightly unstable. Those tiny differences are fatal in robotics because robots physically interact with the world. Sim2Real being difficult does not mean simulation is useless. Simulation is still incredibly valuable: they are cheap, safe, scalable and reproducible. A better way to think about simulation is: Simulation is the training ground, not the final battlefield. Modern Sim2Real methods usually combine multiple approaches: making simulation more realistic, adding domain randomization, randomizing lighting, friction, object positions, and sensor noise, fine-tuning with real-world data. The goal is not to make the robot adapt to one perfect virtual world. The goal is to make the robot robust enough to survive an imperfect real one. The most important lesson in robotics is: Success in simulation is only the first step. The real test begins when the robot touches the real world. Video Credit: Kevin Zakka

Eren Chen

24,920 views • 1 month ago

To push self-driving into situations wilder than reality, we built a neural network world simulator that can create entirely synthetic worlds for the Tesla to drive in. Video below is fully generated & not a real video
1:15

To push self-driving into situations wilder than reality, we built a neural network world simulator that can create entirely synthetic worlds for the Tesla to drive in. Video below is fully generated & not a real video

Tesla

1,697,597 views • 7 months ago