Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

wenlong_huang's profile picture

Wenlong Huang

5,750 subscribers

293,910 Aufrufe • vor 3 Jahren •via X (Twitter)

KunstWissenschaft & TechnologieBildung
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

Data is key for generalization, but robot data is scarce and expensive. Instead of training policies on labeled data, VoxPoser uses LLM+VLM to compose 3D value maps using generated code. Then 6-DoF actions are synthesized by motion planners, all w/o any training or primitives.

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

Given free-form instructions + RGB-D obs, LLM orchestrates perception calls to VLM and array operations to assign continuous values to voxel map, showing *where to act* and *how to act*. It also parametrizes rotation, velocity, and gripper actions for a complete SE(3) trajectory!

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

We verified VoxPoser in everyday manipulation tasks in the wild, including articulated and deformable object manipulation. All the results here are synthesized with zero-shot execution.

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

Just toss your objects too! VoxPoser is robust to disturbances because it replans actions in *real-time* with visual feedback. The 3D value maps are always updated with latest observations, allowing robot to recover from unexpected errors.

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

LLMs show emergent abilities at scale – same applies to VoxPoser, but on physical behaviors! It can conduct physics experiments, have behavioral commonsense, listen to your fine-grained correction, come up with multi-step visual program, and more.

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

For more, check out 👇 Project site: Walkthrough video: Paper: Work done w/ @chenwang_j , @RuohanZhang76 , @YunzhuLiYZ , @jiajunwu_cs, and @drfeifei at @StanfordSVL & @StanfordAILab.

Profilbild von Wenlong Huang
Wenlong Huangvor 2 Jahren

@RuohanZhang76 @YunzhuLiYZ @jiajunwu_cs @drfeifei @StanfordSVL @StanfordAILab Excited to share that we have open-sourced the code based on RLBench: We will also be presenting VoxPoser at CoRL next week as an oral presentation. See you in Atlanta! #CoRL @corl_conf

Profilbild von Brett Adcock
Brett Adcockvor 3 Jahren

This is great

Profilbild von Wenlong Huang
Wenlong Huangvor 3 Jahren

Thank you Brett!

Profilbild von DK Xu
DK Xuvor 3 Jahren

interesting work!!

Ähnliche Videos

What structural task representation enables multi-stage, in-the-wild, bimanual, reactive manipulation? Introducing ReKep: LVM to label keypoints & VLM to write keypoint-based constraints, solve w/ optimization for diverse tasks, w/o task-specific training or env models. 🧵👇
2:10

What structural task representation enables multi-stage, in-the-wild, bimanual, reactive manipulation? Introducing ReKep: LVM to label keypoints & VLM to write keypoint-based constraints, solve w/ optimization for diverse tasks, w/o task-specific training or env models. 🧵👇

Wenlong Huang

191,102 Aufrufe • vor 2 Jahren

Today, we present a step-change in robotic AI Sunday. Introducing ACT-1: A frontier robot foundation model trained on zero robot data. - Ultra long-horizon tasks - Zero-shot generalization - Advanced dexterity 🧵->
4:09

Today, we present a step-change in robotic AI Sunday. Introducing ACT-1: A frontier robot foundation model trained on zero robot data. - Ultra long-horizon tasks - Zero-shot generalization - Advanced dexterity 🧵->

Tony Zhao

2,055,633 Aufrufe • vor 9 Monaten

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)
1:15

How to use simulation data for real-world robot manipulation? We present sim-and-real co-training, a simple recipe for manipulation. We demonstrate that sim data can significantly enhance real-world performance, even with notable differences between the sim and the real. (1/n)

Zhenyu Jiang

44,440 Aufrufe • vor 1 Jahr

How can we move beyond static-arm lab setups and learn robot policies in our messy homes? We introduce HoMeR, an imitation learning agent for in-the-wild mobile manipulation. 🧵1/8
1:48

How can we move beyond static-arm lab setups and learn robot policies in our messy homes? We introduce HoMeR, an imitation learning agent for in-the-wild mobile manipulation. 🧵1/8

Priya Sundaresan

43,107 Aufrufe • vor 1 Jahr

Field AI raised $405M in funding and introduced Field Foundation Models These models are designed to grapple with uncertainties and the physical constraints of the real world, enabling safe robot behaviors when navigating in new environments
0:24

Field AI raised $405M in funding and introduced Field Foundation Models These models are designed to grapple with uncertainties and the physical constraints of the real world, enabling safe robot behaviors when navigating in new environments

Brett Adcock

15,136 Aufrufe • vor 1 Jahr

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲
0:29

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲

Yang Gao

125,057 Aufrufe • vor 1 Jahr

We released τ₀-VLA, a robot foundation model built for long-horizon real-world manipulation. Robots can now clean rooms, cook meals, make drinks, collect laundry, and organize objects in autonomous episodes lasting up to 12 minutes. 🧵
3:32

We released τ₀-VLA, a robot foundation model built for long-horizon real-world manipulation. Robots can now clean rooms, cook meals, make drinks, collect laundry, and organize objects in autonomous episodes lasting up to 12 minutes. 🧵

Jianlan Luo

51,946 Aufrufe • vor 1 Monat

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.
3:26

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

91,623 Aufrufe • vor 10 Monaten

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇
1:06

Training robots for the open world needs diverse data But collecting robot demos in the wild is hard! Presenting DexWild 🙌🏕️ Human data collection system that works in diverse environments, without robots 💪🦾 Human + Robot Cotraining pipeline that unlocks generalization 🧵👇

Tony Tao

97,703 Aufrufe • vor 1 Jahr

Can robot foundation models scale? Xiaomi-Robotics-1 explores this question with over 100,000 hours of real-world manipulation data. Pre-trained on large-scale real-world trajectories and post-trained with cross-embodiment robot data, Xiaomi-Robotics-1 demonstrates consistent scaling across data and model size, strong generalization in unseen environments, and efficient adaptation to new tasks. 🔗 #Robotics #EmbodiedAI #FoundationModels #RobotLearning #XiaomiAI #XiaomiRobotics
0:29

Can robot foundation models scale? Xiaomi-Robotics-1 explores this question with over 100,000 hours of real-world manipulation data. Pre-trained on large-scale real-world trajectories and post-trained with cross-embodiment robot data, Xiaomi-Robotics-1 demonstrates consistent scaling across data and model size, strong generalization in unseen environments, and efficient adaptation to new tasks. 🔗 #Robotics #EmbodiedAI #FoundationModels #RobotLearning #XiaomiAI #XiaomiRobotics

Xiaomi Tech

22,675 Aufrufe • vor 1 Monat

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)
0:33

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!
54:52

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

27,877 Aufrufe • vor 7 Monaten

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM
0:53

Robotics in the age of FMs, cloud and Starlink. We've been rethinking robot AI — cloud-first, LLM-first approach. Fast to deploy. Modular. Data-centric. Built for long-horizon reasoning. Paper: Blog: 🧵 #Robotics #AI #LLM #VLM

Ashish Kapoor

10,185 Aufrufe • vor 11 Monaten

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!
1:29

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,827 Aufrufe • vor 1 Jahr

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.
0:25

The Real-to-Sim part of R2S2R transforms physical robots, sensors, environments, objects, and interactions into simulations that preserve task-relevant observations and dynamics - not only how the world looks, but how it acts when the robot interacts with it. These results set a new bar for sim-real alignment in robot manipulation.

Fei-Fei Li

25,720 Aufrufe • vor 1 Monat

Cortex AI (CortexAI) produces the world's most diverse egocentric and robot dataset, captured in real workplace environments. Leading research labs use Cortex AI to collect data and deploy robotics foundation models in real-world settings.
0:36

Cortex AI (CortexAI) produces the world's most diverse egocentric and robot dataset, captured in real workplace environments. Leading research labs use Cortex AI to collect data and deploy robotics foundation models in real-world settings.

Y Combinator

16,336 Aufrufe • vor 9 Monaten

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇
0:25

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Lerrel Pinto

56,591 Aufrufe • vor 2 Jahren

Cross-embodied robot policies hold the promise of one policy to control all robots. But how far does transfer go? In new work, we study positive transfer between *manipulation* & *navigation* and show that nav data helps manipulation, and vice versa! 🧵 👇
0:53

Cross-embodied robot policies hold the promise of one policy to control all robots. But how far does transfer go? In new work, we study positive transfer between *manipulation* & *navigation* and show that nav data helps manipulation, and vice versa! 🧵 👇

Sergey Levine

42,721 Aufrufe • vor 2 Jahren

🔥 #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora — the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. 🦾 Dual Arms 🖐️ Dual Hands 🎯 36 DoF Control 🌍 Open Source Trained on: • 100K simulated trajectories • 10K real-world demonstrations Dexora achieves: ✓ 90%+ success on basic manipulation ✓ Strong dexterous manipulation performance ✓ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. 🎥 Demos: 📄 Paper:
2:58

🔥 #ICRA2026 Best Paper Finalist The era of "robot VLA = single-arm gripper" is ending. Introducing Dexora — the first open-source Vision-Language-Action system for dual-arm, dual-hand, 36-DoF dexterous manipulation. 🦾 Dual Arms 🖐️ Dual Hands 🎯 36 DoF Control 🌍 Open Source Trained on: • 100K simulated trajectories • 10K real-world demonstrations Dexora achieves: ✓ 90%+ success on basic manipulation ✓ Strong dexterous manipulation performance ✓ Cross-embodiment generalization Our key hypothesis: Train on the hardest embodiment. Transfer to simpler robots later. Instead of scaling up gripper policies, we train directly in the most expressive action space and project downward to simpler embodiments. This may be a practical path toward universal robot controllers. 🎥 Demos: 📄 Paper:

Hao Zhao

17,474 Aufrufe • vor 3 Monaten