正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Excited to release FAST, our new robot action tokenizer! 🤖 Some highlights: - Simple autoregressive VLAs match diffusion VLA performance - Trains up to 5x faster - Works on all robot datasets we tested - First VLAs that work out-of-the-box in new environments! 🧵/

Karl Pertsch

4,386 subscribers

90,700 次观看 • 1 年前 •via X (Twitter)

教育艺术科学技术

Anya Rossi• Live Now

Private livecam show

11 条评论

Karl Pertsch 的头像

Karl Pertsch1 年前

The key idea: FAST compresses actions before training on them. This removes redundancy & makes autoregressive VLA training on high-frequency tasks possible, where models like OpenVLA failed. We use the discrete cosine transform for compressing actions (also used by eg JPEG). 2/

Karl Pertsch 的头像

Karl Pertsch1 年前

With FAST, we scale autoregressive VLA training to pi0 scale, and we can solve some pretty complex robot tasks, simply via next token prediction! The best part: in our experiments, pi0+FAST converges 5x faster than diffusion pi0! Days instead of weeks of training! 🎉 3/

Karl Pertsch 的头像

Karl Pertsch1 年前

My favorite result: with FAST we can finally train VLAs on the DROID dataset & they work zero-shot in many scenes! Below is the same policy controlling robots at Berkeley, Stanford and UW. Just point a camera at the scene, type out an instruction, et voila! 4/

Karl Pertsch 的头像

Karl Pertsch1 年前

We are releasing a FAST tokenizer we pre-trained on 1M real robot action sequences. In our tests it works well across all kind of robots — and it’s all on HuggingFace! Happy VLA training! :) 5/

Karl Pertsch 的头像

Karl Pertsch1 年前

I am very excited about FAST, because (1) it makes VLA training really easy, even on complex tasks, and (2) with FAST it’s trivial to interleave non-robot data in VLA training (web data, subgoals, video prediction etc), it’s all just tokens! Lots of things to explore! :) 6/

Karl Pertsch 的头像

Karl Pertsch1 年前

Please find more details about FAST in our paper! Thanks to @KyleStachowicz and many colleagues @physical_int who helped with this project! Paper: Website:

ARK Electronics 的头像

ARK Electronics2 年前

Excited about the latest tech for your drone product? Our NDAA-compliant, US-made flight controllers are designed to accelerate your path to market and provide a solid platform for developing your autonomous software. Check them out! #Drones #UAV #UAS #Robotics #MadeInUSA

Ted Xiao 的头像

Ted Xiao1 年前

Neat idea, great work @KarlPertsch and co!

Karl Pertsch 的头像

Karl Pertsch1 年前

Thanks Ted! :)

Cheng Chi 的头像

Cheng Chi1 年前

Really cool paper! Congrats Karl!

Karl Pertsch 的头像

Karl Pertsch1 年前

Thanks Cheng! :)

相关视频

Excited to introduce mimic-video, a new class of Video-Action Model that achieves 10x better data efficiency 📉and trains ⏩ 2x faster than standard VLAs!

Excited to introduce mimic-video, a new class of Video-Action Model that achieves 10x better data efficiency 📉and trains ⏩ 2x faster than standard VLAs!

Oier Mees

21,844 次观看 • 8 个月前

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Lerrel Pinto

42,576 次观看 • 1 年前

Embodied chain of thought (ECoT) is a powerful tool to get VLAs to think through problems, but why does it work? In our new work, we analyze various lightweight ECoT-like strategies, including co-training, to see what is the "minimal" amount of reasoning that can boost VLAs 🧵👇

Embodied chain of thought (ECoT) is a powerful tool to get VLAs to think through problems, but why does it work? In our new work, we analyze various lightweight ECoT-like strategies, including co-training, to see what is the "minimal" amount of reasoning that can boost VLAs 🧵👇

Sergey Levine

22,424 次观看 • 1 年前

Real-time inference is a big challenge for VLAs. We’ve been working on a way to amortize inference delays in π0.5. Our new Real-Time Chunking (RTC) method speeds up π0.5 by allowing the robot to “think” while it’s moving, which makes it quite a bit faster! 🧵👇

Real-time inference is a big challenge for VLAs. We’ve been working on a way to amortize inference delays in π0.5. Our new Real-Time Chunking (RTC) method speeds up π0.5 by allowing the robot to “think” while it’s moving, which makes it quite a bit faster! 🧵👇

Sergey Levine

73,688 次观看 • 1 年前

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

Yinghao Xu

46,962 次观看 • 6 个月前

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Lerrel Pinto

56,591 次观看 • 1 年前

Meet our AI-powered robot that’s ready to play table tennis. 🤖🏓 It’s the first agent to achieve amateur human level performance in this sport. Here’s how it works. 🧵

Meet our AI-powered robot that’s ready to play table tennis. 🤖🏓 It’s the first agent to achieve amateur human level performance in this sport. Here’s how it works. 🧵

Google DeepMind

785,170 次观看 • 2 年前

Generative models (diffusion/flow) are taking over robotics 🤖. But do we really need to model the full action distribution to control a robot? We suspected the success of Generative Control Policies (GCPs) might be "Much Ado About Noising." We rigorously tested the myths. 🧵👇

Generative models (diffusion/flow) are taking over robotics 🤖. But do we really need to model the full action distribution to control a robot? We suspected the success of Generative Control Policies (GCPs) might be "Much Ado About Noising." We rigorously tested the myths. 🧵👇

Chaoyi Pan

112,126 次观看 • 8 个月前

Very happy to share our new work APRL (+ open-source code release)! The important step forward we took here is enabling the robot to keep improving with more data—walking faster and adapting to new situations—where prior work saturates.

Very happy to share our new work APRL (+ open-source code release)! The important step forward we took here is enabling the robot to keep improving with more data—walking faster and adapting to new situations—where prior work saturates.

Laura Smith

48,024 次观看 • 2 年前

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Soroush Nasiriany

27,495 次观看 • 1 年前

Introducing GEN-1. Our latest milestone in scaling robot learning. We believe it to be the first general-purpose AI model to master simple physical tasks. 99% success rates, 3x faster speeds, adapts in real time to unexpected scenarios, w/ only 1 hour of robot data. More🧵👇

Introducing GEN-1. Our latest milestone in scaling robot learning. We believe it to be the first general-purpose AI model to master simple physical tasks. 99% success rates, 3x faster speeds, adapts in real time to unexpected scenarios, w/ only 1 hour of robot data. More🧵👇

Generalist

382,267 次观看 • 4 个月前

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

The Humanoid Hub

35,204 次观看 • 6 个月前

1/n: We are excited to share that our paper on Chroma, a general purpose diffusion model for proteins, is out today in nature! A couple of my favorite highlights in the 🧵below 👇

1/n: We are excited to share that our paper on Chroma, a general purpose diffusion model for proteins, is out today in nature! A couple of my favorite highlights in the 🧵below 👇

Andrew Beam

362,102 次观看 • 2 年前

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 次观看 • 1 年前

We took a robot to RSS in LA running our new Gemini Robotics On-Device VLA model. People interacted with the model with new objects and instructions in a brand new environment and the results were amazing!

Ayzaan Wahid

19,702 次观看 • 1 年前

We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵

We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵

Google DeepMind

819,664 次观看 • 1 年前

Tiff In Tech (600K+ on YouTube) just dropped a new video: Robots Are ‘Doomed to Succeed’ and That’s a Problem. Excited to see OpenMind featured in the conversation, highlighting the missing gaps in physical AI we solve: - Robot understanding of human intent - Building real social intelligence in robots - Operating systems that work across all robot manufacturers and form-factors Check out the full video below👇

Tiff In Tech (600K+ on YouTube) just dropped a new video: Robots Are ‘Doomed to Succeed’ and That’s a Problem. Excited to see OpenMind featured in the conversation, highlighting the missing gaps in physical AI we solve: - Robot understanding of human intent - Building real social intelligence in robots - Operating systems that work across all robot manufacturers and form-factors Check out the full video below👇

OpenMind

11,197 次观看 • 4 个月前

We’ve all seen humanoid robots doing backflips and dance routines for years. But if you ask them to climb a few stairs in the real world, they stumble! We took our robot on a walk around town to environments that it hadn’t seen before. Here’s how it works🧵⬇️

We’ve all seen humanoid robots doing backflips and dance routines for years. But if you ask them to climb a few stairs in the real world, they stumble! We took our robot on a walk around town to environments that it hadn’t seen before. Here’s how it works🧵⬇️

Skild AI

271,038 次观看 • 1 年前

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Jason Ma

98,090 次观看 • 1 年前

Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands. This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:

Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands. This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:

Kevin Zakka

259,990 次观看 • 3 年前