Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

This week on Training Data: robots! I really admire @Physical_int for their open publishing spirit. Karol Hausman and Tobias Springenberg joined me and Alfred Lin to chat about pi*0.6, learning from experience, long-horizon robot performance, and more.

Sonya Huang 🐥

20,835 subscribers

12,093 Aufrufe • vor 6 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

LingBot-VLA 2.0 is an impressive new embodied model. Open source and is trained across diverse robot configurations, from single-arm robots to humanoid platforms. It packs 60K hours of curated robot and human data into one generalist policy. It improves robots on difficult long-horizon tasks. Great release by Robbyant.

LingBot-VLA 2.0 is an impressive new embodied model. Open source and is trained across diverse robot configurations, from single-arm robots to humanoid platforms. It packs 60K hours of curated robot and human data into one generalist policy. It improves robots on difficult long-horizon tasks. Great release by Robbyant.

elvis

10,464 Aufrufe • vor 14 Tagen

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes even*cheaper* with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

So I heard we need more data for robot learning :) Purely real world teleop is expensive and slow, making large scale data collection challenging. I’ve been excited about getting more data into robot learning, going beyond just real-world teleop data. To this end, we’ve been scaling up data generation with RL in realistic simulations generated on the fly from crowdsourced videos. Enables realistic data collection, much more cheaply than purely real world teleop. Importantly, data collection becomes evencheaper with more environments, allowing training with over 100x more data. Transfers to real robots for generalizable manipulation. A 🧵 (1/N)

Abhishek Gupta

13,350 Aufrufe • vor 1 Jahr

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

Real-world robot data is expensive and slow to collect, creating a major challenge for humanoid development. 🤖 The NVIDIA GR00T N1.6 open vision language action model is pre-trained on a diverse mix of data, including thousands of hours of Stanford Vision and Learning Lab’s BEHAVIOR simulation data, which covers long-horizon everyday manipulation tasks. This diverse training is the key to robust cross-embodiment performance and real-world adaptability. 🌍 Read the blog 🔗

NVIDIA Robotics

13,456 Aufrufe • vor 6 Monaten

Chelsea Finn (Chelsea Finn) on building general-purpose robotics, and bringing intelligence into the physical world. At AI Startup School in San Francisco. 00:00 - General Purpose Robots 00:11 - Challenges in Robotics Applications 00:57 - Physical Intelligence: A New Approach 01:47 - Learning from Language Models 02:08 - Data Sources for Training Robots 03:32 - Training with Real-World Data 04:39 - Initial Successes and Challenges 09:10 - Breakthrough in Robot Training 11:03 - Improving Performance 15:43 - Expanding Capabilities 17:34 - Robots in Unseen Environments 25:54 - Handling Open-Ended Prompts 29:36 - Evaluating Robot Performance 30:03 - Future Directions and Challenges 31:27 - Audience Q&A

Chelsea Finn (Chelsea Finn) on building general-purpose robotics, and bringing intelligence into the physical world. At AI Startup School in San Francisco. 00:00 - General Purpose Robots 00:11 - Challenges in Robotics Applications 00:57 - Physical Intelligence: A New Approach 01:47 - Learning from Language Models 02:08 - Data Sources for Training Robots 03:32 - Training with Real-World Data 04:39 - Initial Successes and Challenges 09:10 - Breakthrough in Robot Training 11:03 - Improving Performance 15:43 - Expanding Capabilities 17:34 - Robots in Unseen Environments 25:54 - Handling Open-Ended Prompts 29:36 - Evaluating Robot Performance 30:03 - Future Directions and Challenges 31:27 - Audience Q&A

Y Combinator

129,723 Aufrufe • vor 1 Jahr

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

We discovered an emergent property of VLAs like π0/π0.5/π0.6: as we scale up pre-training, the model learns to align human videos and robot data! This gives us a simple way to leverage human videos. Once π0.5 knows how to control robots, it can naturally learn from human video.

Physical Intelligence

1,184,022 Aufrufe • vor 7 Monaten

We asked Angus about the future of robotics and AI training. “I built an industrial-grade kinematic solver for 6 degrees of freedom motion, with full path planning and joint control." "You can stream AI model outputs into it, or run it like a traditional industrial robot." "The benefit of that is all of that information you can stream and record from those joints… you can use that as the training data." "Frankly, moving a robot with a lever is really crap data for training robots. But this gives you smooth, high-quality motion, without needing two robots.”

We asked Angus about the future of robotics and AI training. “I built an industrial-grade kinematic solver for 6 degrees of freedom motion, with full path planning and joint control." "You can stream AI model outputs into it, or run it like a traditional industrial robot." "The benefit of that is all of that information you can stream and record from those joints… you can use that as the training data." "Frankly, moving a robot with a lever is really crap data for training robots. But this gives you smooth, high-quality motion, without needing two robots.”

TBPN

18,811 Aufrufe • vor 1 Jahr

Robotics has changed dramatically over the last eight years. Ted Xiao has been involved in the cutting edge of robot learning through this period, spending those eight years at Google Brain/Google Deepmind. And he’s identified three eras of robot learning. These eras are: - The Era of Existence Proofs - trying different methods like QT-Opt, on-robot RL - The Era of Foundation Models - transitioning to data collection and clean objectives (i.e. supervised learning) - The Era of Scaling - orders of magnitude more data and larger models, enabling reasoning, long-horizon actions, and cross-embodiment transfer Watch Episode 78 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan to learn more!

Robotics has changed dramatically over the last eight years. Ted Xiao has been involved in the cutting edge of robot learning through this period, spending those eight years at Google Brain/Google Deepmind. And he’s identified three eras of robot learning. These eras are: - The Era of Existence Proofs - trying different methods like QT-Opt, on-robot RL - The Era of Foundation Models - transitioning to data collection and clean objectives (i.e. supervised learning) - The Era of Scaling - orders of magnitude more data and larger models, enabling reasoning, long-horizon actions, and cross-embodiment transfer Watch Episode 78 of RoboPapers, with Michael Cho - Rbt/Acc and Jiafei Duan to learn more!

RoboPapers

36,520 Aufrufe • vor 2 Monaten

Almost every Data episode is about him learning about a new life experience and this time on Star Trek TNG Data learns to understand suicide. I love the immediate concern Geordi has for Data when he brings up having a similar experience in the past :(

Almost every Data episode is about him learning about a new life experience and this time on Star Trek TNG Data learns to understand suicide. I love the immediate concern Geordi has for Data when he brings up having a similar experience in the past :(

Yaffa the Lesbian Witch🖖

31,541 Aufrufe • vor 5 Monaten

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Jason Ma

98,090 Aufrufe • vor 1 Jahr

I have Post-Doc, PhD, and Research Assistant positions available, for a new project working with me on dexterous robot learning! Come and join me, my team, and our robots, at Imperial College London! 🤖🧑‍💻🇬🇧💂🤖 See for more info. Please share/retweet!

I have Post-Doc, PhD, and Research Assistant positions available, for a new project working with me on dexterous robot learning! Come and join me, my team, and our robots, at Imperial College London! 🤖🧑‍💻🇬🇧💂🤖 See for more info. Please share/retweet!

Edward Johns

25,053 Aufrufe • vor 1 Jahr

Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence Tutor is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz, and Joe McCalmon, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Collecting robot data at scale is key to deploying working manipulation policies, and the team from Tutor Intelligence Tutor is here to tell us about how to accomplish it. Their new announcement: a massive, 100-robot “data factory,” with a behind-the-scenes look at how to build a teleoperation platform and how to make robots and policies that are useful for their customers. Tutor Intelligence is a full-stack robotics company: they build robot arms, they sell robot arms, they write the software and they train neural networks. Josh Gruenstein, Jesse Michel, Shiraz, and Joe McCalmon, and Joe McCalmon join us to tell us more about how they scale both teleop data and human interventions from their teleoperators in order to train the policies they need. Watch Episode #85 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

36,041 Aufrufe • vor 1 Monat

"I think last year was really a learning experience for me, and this year I'm ready to blossom." Alex Karaban joined Bill Raftery after UConn Men's Basketball's close win over No. 7 ranked BYU.

"I think last year was really a learning experience for me, and this year I'm ready to blossom." Alex Karaban joined Bill Raftery after UConn Men's Basketball's close win over No. 7 ranked BYU.

FOX College Hoops

24,277 Aufrufe • vor 8 Monaten

Robots Using Their Feet to Manipulate Objects ETH Zurich researchers have advanced "pedipulation," where legged robots use their feet for manipulation instead of robotic arms. Their new approach trains robots to navigate and manipulate objects while avoiding static and dynamic obstacles. Tested on the ANYmal quadruped, the method enables precise foot control in complex, unseen environments after minimal training in simulation. Will this research open up new possibilities for efficient, obstacle-aware robot manipulation?

Robots Using Their Feet to Manipulate Objects ETH Zurich researchers have advanced "pedipulation," where legged robots use their feet for manipulation instead of robotic arms. Their new approach trains robots to navigate and manipulate objects while avoiding static and dynamic obstacles. Tested on the ANYmal quadruped, the method enables precise foot control in complex, unseen environments after minimal training in simulation. Will this research open up new possibilities for efficient, obstacle-aware robot manipulation?

Circuit

11,802 Aufrufe • vor 1 Jahr

A few days ago we got in YC W26, and here is we are working on. Building hardware is hard, but I really like a quote from Yuke Zhu: “People who are really serious about robot learning should make their own robot hardware.”

A few days ago we got in YC W26, and here is we are working on. Building hardware is hard, but I really like a quote from Yuke Zhu: “People who are really serious about robot learning should make their own robot hardware.”

Quanting Xie

125,066 Aufrufe • vor 6 Monaten

sophia talking about having adhd meant so much to me :( hearing someone who inspires me and that i admire and look up to so much openly talk about having the same struggles as i do, made me feel really seen and represented. seeing her be open about it honestly comforted me more than i can explain 🤍

sophia talking about having adhd meant so much to me :( hearing someone who inspires me and that i admire and look up to so much openly talk about having the same struggles as i do, made me feel really seen and represented. seeing her be open about it honestly comforted me more than i can explain 🤍

daily sophia

20,957 Aufrufe • vor 2 Monaten

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 Aufrufe • vor 1 Jahr

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,334 Aufrufe • vor 1 Monat

Pliny the Liberator asked "what is on the horizon?" Well, yesterday I visited Silicon Valley's InOrbit robot space to learn more about what's next in robotics. This place has a ton of specialized robots that do a variety of things for a variety of different businesses. sierra catalina will be happy, not a humanoid in it yet. But what they really are doing, beyond a community center for the robotics industry, is building an orchestration system for warehouses and factories and we get a good look at that too. Founder/CEO Florian Pestoni gives me a tour. So, Pliny the Liberator 🐉󠅫󠄼󠄿󠅆󠄵󠄐󠅀󠄼󠄹󠄾󠅉󠅭, here's your answer to "what is on the horizon:" More robots. Learn more about InOrbit here:

Pliny the Liberator asked "what is on the horizon?" Well, yesterday I visited Silicon Valley's InOrbit robot space to learn more about what's next in robotics. This place has a ton of specialized robots that do a variety of things for a variety of different businesses. sierra catalina will be happy, not a humanoid in it yet. But what they really are doing, beyond a community center for the robotics industry, is building an orchestration system for warehouses and factories and we get a good look at that too. Founder/CEO Florian Pestoni gives me a tour. So, Pliny the Liberator 🐉󠅫󠄼󠄿󠅆󠄵󠄐󠅀󠄼󠄹󠄾󠅉󠅭, here's your answer to "what is on the horizon:" More robots. Learn more about InOrbit here:

Robert Scoble

69,807 Aufrufe • vor 1 Jahr