Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Punchline: distill world models from simulation to enable fast, stable real-world robot adaptation. Simulation is nearly always wrong. But in Simulation Distillation, we ask a simple question: How do we perform simulation pretraining such that real-world adaptation becomes trivially easy? Let's take a closer look (1/n)

Abhishek Gupta

11,529 subscribers

32,721 görüntüleme • 2 ay önce •via X (Twitter)

Eğitim Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

Zhenyu Jiang

44,310 görüntüleme • 1 yıl önce

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 görüntüleme • 1 yıl önce

🔍 Take a closer look at how your two early game ammo options perform and why the difference matters. Our complex ballistic simulation authentically translates real-world bullet behavior into #GrayZoneWarfare.

🔍 Take a closer look at how your two early game ammo options perform and why the difference matters. Our complex ballistic simulation authentically translates real-world bullet behavior into #GrayZoneWarfare.

Gray Zone Warfare 🔜 Gamescom

28,600 görüntüleme • 7 ay önce

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

Jiafei Duan

36,290 görüntüleme • 1 ay önce

Our world is not what we were taught. We do live in a holographic simulation. The universe is mathematical.

Our world is not what we were taught. We do live in a holographic simulation. The universe is mathematical.

Diana ❤️🇺🇸🙏🐸

18,336 görüntüleme • 3 ay önce

📢 Announcing one of the most exciting works from us this year on **scalable robot policy evaluation through real-to-sim transfer**, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is **orders of magnitude slower** than the development of language agents. We propose a new framework where simulation performance **strongly correlates** with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing **state-of-the-art correlation** between simulation and reality for deformable object manipulation. It provides a **scalable and reproducible evaluation engine** for robot learning. 🌐

📢 Announcing one of the most exciting works from us this year on scalable robot policy evaluation through real-to-sim transfer, moving toward a scalable evaluation engine with structured world models that capture the appearance, geometry, and dynamics of environments involving deformable objects. 🤖 Evaluation remains one of the biggest bottlenecks in building general-purpose robots. Today, robots are still evaluated only in the real world, which is orders of magnitude slower than the development of language agents. We propose a new framework where simulation performance strongly correlates with the real world (r > 0.9), even for deformable objects. The key difference from existing work lies in the correlation between simulation and reality: if a robot model performs better in the digital world, does it also perform better in the real world? This question has long made people hesitant about simulation-based evaluation — especially for deformable objects. We are changing that. Our pipeline achieves effective real-to-sim transfer, establishing state-of-the-art correlation between simulation and reality for deformable object manipulation. It provides a scalable and reproducible evaluation engine for robot learning. 🌐

Yunzhu Li

39,900 görüntüleme • 8 ay önce

From a single image to a full robotics simulation. 360 image → Marble → 3D environment → Robotics sim This is how we scale world creation for embodied AI with Lightwheel, turning lightweight capture into simulation-ready environments in minutes, not weeks.

From a single image to a full robotics simulation. 360 image → Marble → 3D environment → Robotics sim This is how we scale world creation for embodied AI with Lightwheel, turning lightweight capture into simulation-ready environments in minutes, not weeks.

World Labs

45,847 görüntüleme • 6 ay önce

Watch this robot dog learn to walk from scratch in real time! Our new method, APRL, dynamically adjusts exploration constraints to enable fast and performant RL directly in the real world. APRL can also adapt to changes in the terrain. No simulation, no demos. A thread 👇

Watch this robot dog learn to walk from scratch in real time! Our new method, APRL, dynamically adjusts exploration constraints to enable fast and performant RL directly in the real world. APRL can also adapt to changes in the terrain. No simulation, no demos. A thread 👇

Sergey Levine

105,579 görüntüleme • 2 yıl önce

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 görüntüleme • 1 yıl önce

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Robot foundation models are limited by costly real data, while simulation data is plentiful but visually mismatched to reality. We present Point Bridge, a method that enables zero-shot sim-to-real transfer for robot learning with minimal visual alignment.

Siddhant Haldar

19,897 görüntüleme • 5 ay önce

🚨 Roman Yampolsky: "We are DEFINITELY living in a simulation - the probability is pretty close to 100%" 😱 AI safety legend Roman Yampolskiy just broke it down: “I know never to say 100% but pretty close to that.” He’s got the paper too “How to Hack the Simulation.” His big question: Can superintelligent agents jailbreak out of their virtual box? Elon Musk’s one question to AGI? “What’s outside the simulation?” What if this whole reality is just a test… and the real challenge is realizing the world isn’t real? Have you ever felt the glitches? Do you think we can actually escape? What would YOU ask once you broke out? Drop your wildest theory below 👇

🚨 Roman Yampolsky: "We are DEFINITELY living in a simulation - the probability is pretty close to 100%" 😱 AI safety legend Roman Yampolskiy just broke it down: “I know never to say 100% but pretty close to that.” He’s got the paper too “How to Hack the Simulation.” His big question: Can superintelligent agents jailbreak out of their virtual box? Elon Musk’s one question to AGI? “What’s outside the simulation?” What if this whole reality is just a test… and the real challenge is realizing the world isn’t real? Have you ever felt the glitches? Do you think we can actually escape? What would YOU ask once you broke out? Drop your wildest theory below 👇

Interstellar

152,809 görüntüleme • 3 ay önce

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 görüntüleme • 4 ay önce

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

The best way to get robust, high-quality robot performance is through reinforcement learning; but RL in either the real world or a traditional simulation has lots of limitations. Instead, Jiazhi Yang in RISE does RL in a compositional world model. Learn more ->

Chris Paxton

33,766 görüntüleme • 1 ay önce

AI is a massive unlock for medicine because once you can simulate cells/human biology you can fast track cures for almost all diseases. Precision medicine becomes real. But if we can simulate cells then we are not far off from simulating the world Are we going to be running simulations within simulation? How many simulations deep are we?

AI is a massive unlock for medicine because once you can simulate cells/human biology you can fast track cures for almost all diseases. Precision medicine becomes real. But if we can simulate cells then we are not far off from simulating the world Are we going to be running simulations within simulation? How many simulations deep are we?

Andrew Kang

54,207 görüntüleme • 1 yıl önce

I don't believe we live in a simulation like in the movie. I believe we are energy, but not a simulation. The video doesn't look real. But. Draw your own conclusions. My Spanish isn't very good, but this guy is saying something about the simulation failing with free energy, and that's why there are pyramids everywhere to generate free energy. Without cables and with a lot of free energy, the system can overload and the simulation ends in failure. I don't know exactly.

I don't believe we live in a simulation like in the movie. I believe we are energy, but not a simulation. The video doesn't look real. But. Draw your own conclusions. My Spanish isn't very good, but this guy is saying something about the simulation failing with free energy, and that's why there are pyramids everywhere to generate free energy. Without cables and with a lot of free energy, the system can overload and the simulation ends in failure. I don't know exactly.

Bronze Giant

456,269 görüntüleme • 11 ay önce

Roman Yampolskiy says we are probably in a computer simulation run by advanced AI to simulate an interesting time in history and we should try to hack out of it to take over avatar bodies in the "real world"

Tsarathustra

115,135 görüntüleme • 1 yıl önce

Self-learning worked in AlphaGo because the world was a clean rulebook, says Demis Hassabis. Real life has no perfect rulebook, so long-term planning needs a strong “world model” to simulate what happens next. Without that simulation, a robot has to guess in the real world—where mistakes can be costly and irreversible. Better world models are the first missing piece, says Demis Hassabis.

Self-learning worked in AlphaGo because the world was a clean rulebook, says Demis Hassabis. Real life has no perfect rulebook, so long-term planning needs a strong “world model” to simulate what happens next. Without that simulation, a robot has to guess in the real world—where mistakes can be costly and irreversible. Better world models are the first missing piece, says Demis Hassabis.

Wes Roth

15,757 görüntüleme • 5 ay önce

Hunyuan published GameCraft! although we could see for a while now where we are heading, it still blows my mind to see world-simulation models actually take shape i don't think people understand how media is about to radically change

Hunyuan published GameCraft! although we could see for a while now where we are heading, it still blows my mind to see world-simulation models actually take shape i don't think people understand how media is about to radically change

Dreaming Tulpa 🥓👑

98,549 görüntüleme • 1 yıl önce

Wow. This is crazy. A developer trained an AI agent in simulation and deployed it onto a real robotic air hockey table using reinforcement learning. This robot can track the puck with millimeter-level accuracy and react in roughly 20 milliseconds, fast enough to challenge even skilled human players. We’re moving from robots that follow programmed rules to machines that learn strategies in simulation and execute them in the physical world.

Wow. This is crazy. A developer trained an AI agent in simulation and deployed it onto a real robotic air hockey table using reinforcement learning. This robot can track the puck with millimeter-level accuracy and react in roughly 20 milliseconds, fast enough to challenge even skilled human players. We’re moving from robots that follow programmed rules to machines that learn strategies in simulation and execute them in the physical world.

SciTech Era

1,584,241 görüntüleme • 1 ay önce

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Abhishek Gupta

33,689 görüntüleme • 18 gün önce