Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Excited to release FAST, our new robot action tokenizer! 🤖 Some highlights: - Simple autoregressive VLAs match diffusion VLA performance - Trains up to 5x faster - Works on all robot datasets we tested - First VLAs that work out-of-the-box in new environments! 🧵/

Karl Pertsch

4,386 subscribers

90,700 görüntüleme • 1 yıl önce •via X (Twitter)

Eğitim Sanat Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

11 Yorum

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

The key idea: FAST compresses actions before training on them. This removes redundancy & makes autoregressive VLA training on high-frequency tasks possible, where models like OpenVLA failed. We use the discrete cosine transform for compressing actions (also used by eg JPEG). 2/

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

With FAST, we scale autoregressive VLA training to pi0 scale, and we can solve some pretty complex robot tasks, simply via next token prediction! The best part: in our experiments, pi0+FAST converges 5x faster than diffusion pi0! Days instead of weeks of training! 🎉 3/

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

My favorite result: with FAST we can finally train VLAs on the DROID dataset & they work zero-shot in many scenes! Below is the same policy controlling robots at Berkeley, Stanford and UW. Just point a camera at the scene, type out an instruction, et voila! 4/

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

We are releasing a FAST tokenizer we pre-trained on 1M real robot action sequences. In our tests it works well across all kind of robots — and it’s all on HuggingFace! Happy VLA training! :) 5/

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

I am very excited about FAST, because (1) it makes VLA training really easy, even on complex tasks, and (2) with FAST it’s trivial to interleave non-robot data in VLA training (web data, subgoals, video prediction etc), it’s all just tokens! Lots of things to explore! :) 6/

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

Please find more details about FAST in our paper! Thanks to @KyleStachowicz and many colleagues @physical_int who helped with this project! Paper: Website:

ARK Electronics profil fotoğrafı

ARK Electronics2 yıl önce

Excited about the latest tech for your drone product? Our NDAA-compliant, US-made flight controllers are designed to accelerate your path to market and provide a solid platform for developing your autonomous software. Check them out! #Drones #UAV #UAS #Robotics #MadeInUSA

Ted Xiao profil fotoğrafı

Ted Xiao1 yıl önce

Neat idea, great work @KarlPertsch and co!

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

Thanks Ted! :)

Cheng Chi profil fotoğrafı

Cheng Chi1 yıl önce

Really cool paper! Congrats Karl!

Karl Pertsch profil fotoğrafı

Karl Pertsch1 yıl önce

Thanks Cheng! :)

Benzer Videolar

Excited to introduce mimic-video, a new class of Video-Action Model that achieves 10x better data efficiency 📉and trains ⏩ 2x faster than standard VLAs!

Excited to introduce mimic-video, a new class of Video-Action Model that achieves 10x better data efficiency 📉and trains ⏩ 2x faster than standard VLAs!

Oier Mees

21,844 görüntüleme • 7 ay önce

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Lerrel Pinto

42,576 görüntüleme • 1 yıl önce

Embodied chain of thought (ECoT) is a powerful tool to get VLAs to think through problems, but why does it work? In our new work, we analyze various lightweight ECoT-like strategies, including co-training, to see what is the "minimal" amount of reasoning that can boost VLAs 🧵👇

Embodied chain of thought (ECoT) is a powerful tool to get VLAs to think through problems, but why does it work? In our new work, we analyze various lightweight ECoT-like strategies, including co-training, to see what is the "minimal" amount of reasoning that can boost VLAs 🧵👇

Sergey Levine

22,424 görüntüleme • 1 yıl önce

Real-time inference is a big challenge for VLAs. We’ve been working on a way to amortize inference delays in π0.5. Our new Real-Time Chunking (RTC) method speeds up π0.5 by allowing the robot to “think” while it’s moving, which makes it quite a bit faster! 🧵👇

Real-time inference is a big challenge for VLAs. We’ve been working on a way to amortize inference delays in π0.5. Our new Real-Time Chunking (RTC) method speeds up π0.5 by allowing the robot to “think” while it’s moving, which makes it quite a bit faster! 🧵👇

Sergey Levine

73,688 görüntüleme • 1 yıl önce

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

🚀 We’re excited to announce LingBot-VA, a new state-of-the-art robot policy model from Robbyant ! LingBot-VA is built on a causal, autoregressive video-action world model for generalist robot control. Highlights: (1) First unified autoregressive video-action world model for robot control (2) Low-latency inference with a new asynchronous execution pipeline (3) SOTA on RoboTwin (92.9%, firstever > 90%) and LIBERO (98.5%) (4) +20% over π0.5 on challenging real-world long-horizon & high-precision tasks

Yinghao Xu

46,962 görüntüleme • 6 ay önce

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Lerrel Pinto

56,591 görüntüleme • 1 yıl önce

Meet our AI-powered robot that’s ready to play table tennis. 🤖🏓 It’s the first agent to achieve amateur human level performance in this sport. Here’s how it works. 🧵

Meet our AI-powered robot that’s ready to play table tennis. 🤖🏓 It’s the first agent to achieve amateur human level performance in this sport. Here’s how it works. 🧵

Google DeepMind

785,170 görüntüleme • 2 yıl önce

Generative models (diffusion/flow) are taking over robotics 🤖. But do we really need to model the full action distribution to control a robot? We suspected the success of Generative Control Policies (GCPs) might be "Much Ado About Noising." We rigorously tested the myths. 🧵👇

Generative models (diffusion/flow) are taking over robotics 🤖. But do we really need to model the full action distribution to control a robot? We suspected the success of Generative Control Policies (GCPs) might be "Much Ado About Noising." We rigorously tested the myths. 🧵👇

Chaoyi Pan

112,064 görüntüleme • 8 ay önce

Very happy to share our new work APRL (+ open-source code release)! The important step forward we took here is enabling the robot to keep improving with more data—walking faster and adapting to new situations—where prior work saturates.

Very happy to share our new work APRL (+ open-source code release)! The important step forward we took here is enabling the robot to keep improving with more data—walking faster and adapting to new situations—where prior work saturates.

Laura Smith

48,024 görüntüleme • 2 yıl önce

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Soroush Nasiriany

27,495 görüntüleme • 1 yıl önce

Introducing GEN-1. Our latest milestone in scaling robot learning. We believe it to be the first general-purpose AI model to master simple physical tasks. 99% success rates, 3x faster speeds, adapts in real time to unexpected scenarios, w/ only 1 hour of robot data. More🧵👇

Introducing GEN-1. Our latest milestone in scaling robot learning. We believe it to be the first general-purpose AI model to master simple physical tasks. 99% success rates, 3x faster speeds, adapts in real time to unexpected scenarios, w/ only 1 hour of robot data. More🧵👇

Generalist

382,192 görüntüleme • 4 ay önce

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

The Humanoid Hub

35,204 görüntüleme • 6 ay önce

1/n: We are excited to share that our paper on Chroma, a general purpose diffusion model for proteins, is out today in nature! A couple of my favorite highlights in the 🧵below 👇

1/n: We are excited to share that our paper on Chroma, a general purpose diffusion model for proteins, is out today in nature! A couple of my favorite highlights in the 🧵below 👇

Andrew Beam

362,098 görüntüleme • 2 yıl önce

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 görüntüleme • 1 yıl önce

We took a robot to RSS in LA running our new Gemini Robotics On-Device VLA model. People interacted with the model with new objects and instructions in a brand new environment and the results were amazing!

Ayzaan Wahid

19,688 görüntüleme • 1 yıl önce

We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵

We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖 It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵

Google DeepMind

819,664 görüntüleme • 1 yıl önce

Tiff In Tech (600K+ on YouTube) just dropped a new video: Robots Are ‘Doomed to Succeed’ and That’s a Problem. Excited to see OpenMind featured in the conversation, highlighting the missing gaps in physical AI we solve: - Robot understanding of human intent - Building real social intelligence in robots - Operating systems that work across all robot manufacturers and form-factors Check out the full video below👇

Tiff In Tech (600K+ on YouTube) just dropped a new video: Robots Are ‘Doomed to Succeed’ and That’s a Problem. Excited to see OpenMind featured in the conversation, highlighting the missing gaps in physical AI we solve: - Robot understanding of human intent - Building real social intelligence in robots - Operating systems that work across all robot manufacturers and form-factors Check out the full video below👇

OpenMind

11,197 görüntüleme • 4 ay önce

We’ve all seen humanoid robots doing backflips and dance routines for years. But if you ask them to climb a few stairs in the real world, they stumble! We took our robot on a walk around town to environments that it hadn’t seen before. Here’s how it works🧵⬇️

We’ve all seen humanoid robots doing backflips and dance routines for years. But if you ask them to climb a few stairs in the real world, they stumble! We took our robot on a walk around town to environments that it hadn’t seen before. Here’s how it works🧵⬇️

Skild AI

271,038 görüntüleme • 1 yıl önce

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Excited to finally share Generative Value Learning (GVL), my Google DeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

Jason Ma

98,090 görüntüleme • 1 yıl önce

Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands. This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:

Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands. This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:

Kevin Zakka

259,990 görüntüleme • 3 yıl önce