Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Gemini Robotics 2 by Google DeepMind uses sim-only post-training on a mixed-data base model to handle automotive kitting. Contact-rich tasks like these have long been considered one of the hardest challenges in Sim2Real. Gemini Robotics 2 shows simulation can work even here. General robot intelligence needs much more data... show more

Axis Robotics

55,974 subscribers

16,857 просмотров • 1 месяц назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

Austin-based robotics startup Apptronik has unveiled Apollo 2, the latest version of its humanoid robot. Apollo 2 comes in both bipedal and wheeled versions and performs real-world tasks while collecting training data. Apptronik is using this data with Google DeepMind to train Gemini Robotics AI models, making Apollo smarter and more capable.

Austin-based robotics startup Apptronik has unveiled Apollo 2, the latest version of its humanoid robot. Apollo 2 comes in both bipedal and wheeled versions and performs real-world tasks while collecting training data. Apptronik is using this data with Google DeepMind to train Gemini Robotics AI models, making Apollo smarter and more capable.

Space and Technology

11,540 просмотров • 2 месяцев назад

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 просмотров • 8 месяцев назад

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 просмотров • 7 месяцев назад

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

Jason Ma reveals the first true scaling law in robotics: robots trained on 1 million hours of first-person human video predictably get better without ever seeing robot data "Yesterday we announced our new flagship robot foundation model called Dyna-2. It's very significant for the entire field for many different reasons. First of all, it's the first robot foundation model trained on at least one million hours of data. That in itself is a very challenging infrastructure challenge." "What we demonstrated is that even if the one million hours of data is entirely first-person video of humans doing manipulation tasks, we actually see scaling law for transferring to robot embodiments that the model has never seen before." "By training on more human data, we actually see predictable performance improvement on robots. That's a big deal because the biggest challenge facing robot foundation models is that we don't have enough data." "Unlike language models, there are just not enough robotics data out there, and it's very difficult to collect these robot data. Only by passively observing humans doing things can we scale this kind of data, and this is the first time we showed there is a lot of improvement that can transfer." Jason Ma Dyna Robotics

MTS

17,740 просмотров • 1 месяц назад

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning *beyond* just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

So we did a bunch of projects with real world reinforcement learning - but it was often too inefficient to be practical to train tabula rasa. This suggests we need better priors, but acquiring these from on-robot data can often be expensive as well. In our recent work, we show that despite being fundamentally inaccurate, simulation can guide provide a cheap way to guide real-world RL finetuning to be super efficient! We propose Simulation-Guided Fine-Tuning (SGFT) - a simple paradigm for sim2real finetuning that uses simulation to provide reward shaping that accelerates real world RL finetuning beyond just providing an initialization. TLDR: Use value functions from sim to shape rewards for real-world RL, see large sample efficiency improvements 🧵(1/6)

Abhishek Gupta

13,637 просмотров • 1 год назад

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

Excitement about world-action models and robot learning has never been higher — they promise a way to use human egocentric data to train massive robotics models which can provide the “GPT” moment for robotics and unlock general-purpose embodied intelligence. And yet there’s been little concrete demonstration of scaling in robot learning. Dyna Robotics aims to change that, with an in-depth look at how scaling works as they approach 1 million hours of training data. Jason Ma Tianyu Li Anh-Quan Pham and Chet Bhateja joined us to tell us more. They show that as the amount of data they use in pretraining increased, they saw predictable, statistically significant gains on accuracy metrics on held-out data (data not seen during training). They go on to talk about what they learned, and show how this can be applied to many different problems. Watch Episode #99 of RoboPapers, with Michael Cho - Rbt/Acc Chris Paxton Jiafei Duan today to learn more!

RoboPapers

26,004 просмотров • 1 месяц назад

Sundar Pichai on the robotics opportunity: ⦿ He's impressed by recent humanoid progress - so much so that he sometimes has to look closely to tell if a robot video is real or fake. ⦿ Google initially moved too early into the application layer, but now sees the combination of AI and robotics as the “sweet spot.” ⦿ He believes we’re 2–3 years away from a “magical moment” in robotics. ⦿ Google is working on playing a role similar to Android in robotics - one of its internal bets is already focused on supporting robotics manufacturers. ⦿ Gemini robotics models will work very well - how to productize them is still an open question at the company.

Sundar Pichai on the robotics opportunity: ⦿ He's impressed by recent humanoid progress - so much so that he sometimes has to look closely to tell if a robot video is real or fake. ⦿ Google initially moved too early into the application layer, but now sees the combination of AI and robotics as the “sweet spot.” ⦿ He believes we’re 2–3 years away from a “magical moment” in robotics. ⦿ Google is working on playing a role similar to Android in robotics - one of its internal bets is already focused on supporting robotics manufacturers. ⦿ Gemini robotics models will work very well - how to productize them is still an open question at the company.

The Humanoid Hub

87,088 просмотров • 1 год назад

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

Co-training has become a key part of the recipe for training large robotics models; it means that you mix some proportion of real robot data with other data sources, like simulation or egocentric human video data. This is especially important because robotics data tends to lack diversity which can be somewhat compensated for by the inclusion of these other modalities. And yet there has not been a sizable study on what constitute good practices for cotraining until now! We talk to Fanqi Lin and Jose Barreiros about their new work, a massive study which evaluated 89 policies over thousands of rollouts to tell us which forms of co-training were most useful for robotics. Watch episode 70 of RoboPapers, with Michael Cho and Chris Paxton, now!

RoboPapers

19,357 просмотров • 6 месяцев назад

Christine (郡主Christine (✱,✱)), co-founder of Axis Robotics spent eight years ago studying mechanical engineering and working with ABB and KUKA robots before pivoting through stablecoin payments and landing on physical AI data. Axis Robotics has raised $12M to build a distributed training engine where anyone with a browser can teach robots. The network already has 200,000+ contributors and a dataset of 50,000+ robot trajectories across 207 tasks. On today's episode: why noisy human data beats "perfect" AI data, why simulation might be the key to physical AI's GPT moment, and the uncomfortable truth about people getting paid to train the robots that'll eventually replace them. Timestamps: 0:00 Intro: Building the Data Layer Behind Physical AI 1:34 Hot Take: Noisy Data vs. Perfect Data 3:16 Why Simulation Data Will Bring the GPT Moment for Robotics 7:25 From Stablecoin Payments to Robotics Data 10:08 Why Axis Chose Data Over Building Robots 16:10 Live Demo: Training a Robot Arm Like Playing FIFA 24:02 How Contributors Get Paid (Tokens + Stablecoin) 26:42 Inside the $12M Seed Round Pitch 32:08 200K+ Contributors & Viral Growth via Base 38:38 Sim-to-Real: The Living Data Engine 41:34 Token & Web3 Plans for Axis Robotics 43:00 Rapid Fire Questions

Christine (郡主Christine (✱,✱)), co-founder of Axis Robotics spent eight years ago studying mechanical engineering and working with ABB and KUKA robots before pivoting through stablecoin payments and landing on physical AI data. Axis Robotics has raised $12M to build a distributed training engine where anyone with a browser can teach robots. The network already has 200,000+ contributors and a dataset of 50,000+ robot trajectories across 207 tasks. On today's episode: why noisy human data beats "perfect" AI data, why simulation might be the key to physical AI's GPT moment, and the uncomfortable truth about people getting paid to train the robots that'll eventually replace them. Timestamps: 0:00 Intro: Building the Data Layer Behind Physical AI 1:34 Hot Take: Noisy Data vs. Perfect Data 3:16 Why Simulation Data Will Bring the GPT Moment for Robotics 7:25 From Stablecoin Payments to Robotics Data 10:08 Why Axis Chose Data Over Building Robots 16:10 Live Demo: Training a Robot Arm Like Playing FIFA 24:02 How Contributors Get Paid (Tokens + Stablecoin) 26:42 Inside the $12M Seed Round Pitch 32:08 200K+ Contributors & Viral Growth via Base 38:38 Sim-to-Real: The Living Data Engine 41:34 Token & Web3 Plans for Axis Robotics 43:00 Rapid Fire Questions

Pukerainbow 🤮🌈

24,329 просмотров • 5 дней назад

15–30 minutes of real-world robot data. That's now enough to go from sim-to-real failure to working robot. Let’s see… You train a robot in simulation. You deploy it in the real world. It fails. The physics don't match. So you try to fine-tune it with real data, but… you never have enough real data, and the fine-tuning breaks everything the simulation taught it. SimDist fixes this with one key decision: don't transfer the policy. Transfer the world model. Keep the reward and value knowledge from simulation frozen. Only update the part that's actually wrong, how the robot predicts physics. Now the robot doesn't have to relearn the entire task in the real world. It already knows what success looks like. It just needs to correct its understanding of how the real world moves. The part that makes this work: they also trained on failures and recoveries; not just perfect demonstrations. Without that, the planner finds the gaps and exploits them. With it, the robot can tell a good future from a bad one. That's all it needs. Results on peg insertion, table leg assembly, locomotion on slippery and uneven surfaces. Tasks that require precision, force, and quick reaction. Thanks for sharing, Tyler Westenbroek ([Tyler Westenbroek]. Interactive visualization + paper: ——- Weekly robotics and AI insights. Subscribe free:

15–30 minutes of real-world robot data. That's now enough to go from sim-to-real failure to working robot. Let’s see… You train a robot in simulation. You deploy it in the real world. It fails. The physics don't match. So you try to fine-tune it with real data, but… you never have enough real data, and the fine-tuning breaks everything the simulation taught it. SimDist fixes this with one key decision: don't transfer the policy. Transfer the world model. Keep the reward and value knowledge from simulation frozen. Only update the part that's actually wrong, how the robot predicts physics. Now the robot doesn't have to relearn the entire task in the real world. It already knows what success looks like. It just needs to correct its understanding of how the real world moves. The part that makes this work: they also trained on failures and recoveries; not just perfect demonstrations. Without that, the planner finds the gaps and exploits them. With it, the robot can tell a good future from a bad one. That's all it needs. Results on peg insertion, table leg assembly, locomotion on slippery and uneven surfaces. Tasks that require precision, force, and quick reaction. Thanks for sharing, Tyler Westenbroek ([Tyler Westenbroek]. Interactive visualization + paper: ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

14,222 просмотров • 4 месяцев назад

Robot trained entirely in simulation! 🤯 Sudo AI just released #sudo R1. This could be the moment the data bottleneck in robotics finally breaks, a major step forward for getting simulation-trained robots to actually work in the real world. #sudo R1 is a manipulation foundation model that reliably picks objects it has never seen: glass, fabric, metal, irregular shapes. All with zero real-world training data. No tele-op. No demos. No manual labeling. Pure simulation. And picking isn't just any skill, it's the foundation. Almost everything we want robots to do starts with being able to reliably pick something up. Data collection is the bottleneck holding the whole field back. Every new task means hours of tele-op and armies of labelers. If simulation alone can actually deliver production-grade picking, that bottleneck starts to disappear. Really exciting to see! If the results hold, the path to general-purpose robots just got a whole lot shorter. 👇 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Robot trained entirely in simulation! 🤯 Sudo AI just released #sudo R1. This could be the moment the data bottleneck in robotics finally breaks, a major step forward for getting simulation-trained robots to actually work in the real world. #sudo R1 is a manipulation foundation model that reliably picks objects it has never seen: glass, fabric, metal, irregular shapes. All with zero real-world training data. No tele-op. No demos. No manual labeling. Pure simulation. And picking isn't just any skill, it's the foundation. Almost everything we want robots to do starts with being able to reliably pick something up. Data collection is the bottleneck holding the whole field back. Every new task means hours of tele-op and armies of labelers. If simulation alone can actually deliver production-grade picking, that bottleneck starts to disappear. Really exciting to see! If the results hold, the path to general-purpose robots just got a whole lot shorter. 👇 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

25,901 просмотров • 5 месяцев назад

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Synthetic data will provide the next trillion tokens to fuel our hungry models. I'm excited to announce MimicGen: massively scaling up data pipeline for robot learning! We multiply high-quality human data in simulation with digital twins. Using 50,000 training episodes across 18 tasks, multiple simulators, and even in the real-world! The idea is simple: 1. Humans tele-operate the robot to complete a task. It is extremely high-quality but also very slow and expensive. 2. We create a digital twin of the robot and the scene in high-fidelity, GPU-accelerated simulation. 3. We can now move objects around, replace with new assets, and even change the robot hand - basically augment the training data with procedural generation. 4. Export the successful episodes, and feed that to a neural network! You now have an near-infinite stream of data. One of the key reasons that robotics lags far behind other AI fields is the lack of data: you cannot scrape control signals from the internet. They simply don't exist in-the-wild. MimicGen shows the power of synthetic data and simulation to keep our scaling laws alive. I believe this principle apply beyond robotics. We are quickly exhausting the high-quality, real tokens from the web. Artificial intelligence from artificial data will be the way forward. We are big fans of the OSS community. As usual, we open-source everything, including the generated dataset! - Website: - Paper: - Dataset is hosted on HuggingFace (thanks AK!!): - Code: MimicGen is led by Ajay Mandlekar, deep dive in the thread:

Jim Fan

332,238 просмотров • 2 лет назад

What if a dexterous robot could learn a 20+ step chemistry experiment without a single on-robot training demo? Meet TwinDEX: a pair of co-designed, three-finger, nine-DoF dexterous manipulation interface: one wearable for data collection, one for robot deployment. The twinned design shares identical kinematics, contact surfaces, visual appearance, and sensors across collection and deployment — keeping observations and actions aligned end to end. Trained from scratch on only a few hundred wearable demonstrations - with zero on-robot training or intervention data - TwinDEX completed a standardized chemistry experiment involving tool switches, fine force control, and bimanual coordination. Robot-free data showed comparable learning efficiency on the multi-task evaluation, TwinDEX delivered 5.3 times effective throughput than on-robot teleoperation. TwinDEX demonstrates that high-quality robot-free data can fully substitute for on-robot teleoperation data on challenging dexterous tasks — removing the dependency on real-robot hardware that has been the central bottleneck to scaling dexterous manipulation data. This was the proof-it phase. Now comes scale: what emerges at tens of thousands, or millions, of episodes? Watch the demo and read the technical blog: #TwinDEX #Robotics #EmbodiedAI #DexterousManipulation

What if a dexterous robot could learn a 20+ step chemistry experiment without a single on-robot training demo? Meet TwinDEX: a pair of co-designed, three-finger, nine-DoF dexterous manipulation interface: one wearable for data collection, one for robot deployment. The twinned design shares identical kinematics, contact surfaces, visual appearance, and sensors across collection and deployment — keeping observations and actions aligned end to end. Trained from scratch on only a few hundred wearable demonstrations - with zero on-robot training or intervention data - TwinDEX completed a standardized chemistry experiment involving tool switches, fine force control, and bimanual coordination. Robot-free data showed comparable learning efficiency on the multi-task evaluation, TwinDEX delivered 5.3 times effective throughput than on-robot teleoperation. TwinDEX demonstrates that high-quality robot-free data can fully substitute for on-robot teleoperation data on challenging dexterous tasks — removing the dependency on real-robot hardware that has been the central bottleneck to scaling dexterous manipulation data. This was the proof-it phase. Now comes scale: what emerges at tens of thousands, or millions, of episodes? Watch the demo and read the technical blog: #TwinDEX #Robotics #EmbodiedAI #DexterousManipulation

X Square Robot

1,346,260 просмотров • 29 дней назад

Happy to share what I’ve been working on since joining Genesis! GENE-26.5 is a one-of-a-kind, robotics-native multimodal foundation model that learns from diverse, in-the-wild data across modalities and outputs actions enabling a 54-DoF robot system to perform the most dexterous, long-horizon manipulation tasks to date—approaching human-level capability. This is the result of innovations across the full stack—data collection and processing, robot systems, model architecture, training strategies, and scalable evaluation infrastructure.

Happy to share what I’ve been working on since joining Genesis! GENE-26.5 is a one-of-a-kind, robotics-native multimodal foundation model that learns from diverse, in-the-wild data across modalities and outputs actions enabling a 54-DoF robot system to perform the most dexterous, long-horizon manipulation tasks to date—approaching human-level capability. This is the result of innovations across the full stack—data collection and processing, robot systems, model architecture, training strategies, and scalable evaluation infrastructure.

Zu

20,315 просмотров • 4 месяцев назад

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

20,434 просмотров • 9 месяцев назад

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Abhishek Gupta

35,283 просмотров • 3 месяцев назад

“We think we have a competitive advantage.” - Demis Hassibis, Head of Google DeepMind / AI efforts While Anthropic and OpenAI seek to win via training on “text” (written language), Demis Hassabis says Google Gemini is attempting a different path: Multimodal. Gemini aims to be the most effective model in the real world: physics, robotics, voice, visual, sensory, & text. The bet is multimodal unlocks more of what humanity really cares about.

“We think we have a competitive advantage.” - Demis Hassibis, Head of Google DeepMind / AI efforts While Anthropic and OpenAI seek to win via training on “text” (written language), Demis Hassabis says Google Gemini is attempting a different path: Multimodal. Gemini aims to be the most effective model in the real world: physics, robotics, voice, visual, sensory, & text. The bet is multimodal unlocks more of what humanity really cares about.

Compound248 💰

17,731 просмотров • 4 месяцев назад

For decades, we’ve dreamed of robots that can seamlessly step into our world and lend a hand. Today, we take a major stride toward making that dream a reality: Introducing Gemini Robotics 2 from Google DeepMind, the intelligence layer powering the next generation of truly adaptable robots. This major advance unlocks intelligent whole-body control, advanced dexterity, and even multi-robot collaboration 🤯. Ok but... how does a robot actually "think"? Real-world tasks take time and planning. To manage that complexity, our new embodied reasoning model, Gemini Robotics ER 2, acts as the robot’s high-level brain, enhancing the robot’s capabilities to: — Observe the environment — Reason about the actions needed to complete the task — Coordinate with the vision-language-action model to carry out actions — Track progress until the job is done This setup allows robots to execute complex multi-step workflows, self-correct if a step fails, and adapt to completely novel situations. Learn more about Gemini Robotics ER 2 (and our two other brand new models) here:

Google AI

786,668 просмотров • 2 месяцев назад

World Labs co-founder Justin Johnson on how five photos from your phone could teach a robot to work in a room it has never seen: "I think there's a couple different tech trees that people are working on. One is this notion of fully in-context learning. Maybe I've got a robotics foundation model, then I can demonstrate to the robot once how a task should be performed, and that's enough for the robot to figure out that task." "Another version is what we're calling real-to-sim-to-real. Maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment." "Like this studio... You could come in here and take just a couple casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space in particular." "Literally you could do it in five minutes. Take five photos, upload it to Atlas, have it generate a simulation, describe in natural language what kind of task you want the robot to do, have an agent go build that simulation for you, RL fine-tune your general purpose robotics foundation model, and now you've got a robot... that's perfectly adapted to this space." Justin Johnson on MTS

World Labs co-founder Justin Johnson on how five photos from your phone could teach a robot to work in a room it has never seen: "I think there's a couple different tech trees that people are working on. One is this notion of fully in-context learning. Maybe I've got a robotics foundation model, then I can demonstrate to the robot once how a task should be performed, and that's enough for the robot to figure out that task." "Another version is what we're calling real-to-sim-to-real. Maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment." "Like this studio... You could come in here and take just a couple casual videos with your phone or a couple images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space in particular." "Literally you could do it in five minutes. Take five photos, upload it to Atlas, have it generate a simulation, describe in natural language what kind of task you want the robot to do, have an agent go build that simulation for you, RL fine-tune your general purpose robotics foundation model, and now you've got a robot... that's perfectly adapted to this space." Justin Johnson on MTS

a16z

35,947 просмотров • 29 дней назад

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 просмотров • 1 год назад