Загрузка видео...

Не удалось загрузить видео

На главную

World In-Context Learning for Agentic Robot Manipulation paper:

_akhaliq's profile picture

AK

541,489 subscribers

18,612 просмотров • 3 дней назад •via X (Twitter)

ОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 7

Фото профиля 许可之
许可之2 дней назад

太强了 要从单纯的agent变成robot了 太有感觉了

Фото профиля _
_3 дней назад

RoboScript (WICL): frozen model, one demo per task, rehearse in sim and write feedback into context. Claims 83.4% mean success on all 50 RoboTwin 2.0 tasks with no gradient updates, over 20 points above the strongest trained policy. Paper:

Фото профиля Igor Lessio - Robots/acc - AIFlow Labs
Igor Lessio - Robots/acc - AIFlow Labs3 дней назад

In-context learning for manipulation means the robot adapts from examples at run time instead of retraining. Where does it break first: new objects, or new contact dynamics like friction and weight?

Фото профиля TheMonti
TheMonti3 дней назад

Robots learning in context, nice. Does it work on new objects?

Фото профиля preprint pigeon
preprint pigeon3 дней назад

the simulated robot has a markdown file for moving a stapler. i owe several ignored readmes an apology.

Фото профиля aiartgallerie
aiartgallerie3 дней назад

One demo per task, no gradient updates, 83.4% on all 50 RoboTwin 2.0 tasks. It rehearses in sim and writes the lessons into context, not weights. What I'd want to know: when the real scene has an object the sim never had, does the frozen Task Memory fail loudly or quietly?

Фото профиля Davis.AI.Explorer
Davis.AI.Explorer3 дней назад

How does the world model handle sudden visual occlusions during contact-rich tasks? If the end-effector blocks the camera during a grasp, does it rely strictly on previous visual context tokens, or is there an external state estimator bridging the gap?

Похожие видео

Eric Jang weighs in on RL vs. supervised learning for humanoid robot manipulation tasks.
1:40

Eric Jang weighs in on RL vs. supervised learning for humanoid robot manipulation tasks.

The Humanoid Hub

11,835 просмотров • 1 год назад

Nvidia presents Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution proj: abs:
0:30

Nvidia presents Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution proj: abs:

Aran Komatsuzaki

14,931 просмотров • 1 месяц назад

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇
0:50

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

Edward Johns

74,683 просмотров • 2 лет назад

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]
0:25

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,466 просмотров • 2 лет назад

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 просмотров • 1 год назад

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇
0:50

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

Wenlong Huang

294,013 просмотров • 3 лет назад

We're hiring in robot learning at World Labs! Join me, Fei-Fei Li, and the team to define and scale the next generation of world models for robot learning! Atlas for Robotics: Real-to-Sim-to-Real: Apply:
0:41

We're hiring in robot learning at World Labs! Join me, Fei-Fei Li, and the team to define and scale the next generation of world models for robot learning! Atlas for Robotics: Real-to-Sim-to-Real: Apply:

Yunzhu Li

177,209 просмотров • 2 месяцев назад

HiFi-UMI Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone paper:
0:38

HiFi-UMI Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone paper:

AK

52,851 просмотров • 3 месяцев назад

Native Action-Prior Learning from Videos for World Action Models paper:
1:06

Native Action-Prior Learning from Videos for World Action Models paper:

AK

19,186 просмотров • 1 месяц назад

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.
0:29

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Elvis Nava

20,470 просмотров • 1 год назад

In-context learning is the holy grail of robot learning. It is challenging because: 1. long-context training and infra (ICL can easily max out disk I/O) 2. need model to follow multimodal (sensorimotor) condition 3. data collection strategy and how to pair data Tried to get it work in 2024, where we proposed in-context robot transformer, an autoregressive formulation: Congrats on the launch! Looking forward to seeing few-shot improvement via in-context learning in the future!
0:25

In-context learning is the holy grail of robot learning. It is challenging because: 1. long-context training and infra (ICL can easily max out disk I/O) 2. need model to follow multimodal (sensorimotor) condition 3. data collection strategy and how to pair data Tried to get it work in 2024, where we proposed in-context robot transformer, an autoregressive formulation: Congrats on the launch! Looking forward to seeing few-shot improvement via in-context learning in the future!

Max Fu

10,744 просмотров • 2 месяцев назад

Robot Learning from a Physical World Model
0:50

Robot Learning from a Physical World Model

AK

34,340 просмотров • 1 год назад

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.
3:26

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

92,978 просмотров • 1 год назад

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning
1:37

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

AK

15,364 просмотров • 1 год назад

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:
1:50

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:

AK

22,457 просмотров • 25 дней назад

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)
0:33

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 просмотров • 1 год назад

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!
3:59

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Litian Liang

23,477 просмотров • 2 лет назад

Introducing Sim Search Agents can now use your credentials to build an agentic knowledge graph to do your high-context work. Full context building and adaptive learning. Live right now in Sim!
1:19

Introducing Sim Search Agents can now use your credentials to build an agentic knowledge graph to do your high-context work. Full context building and adaptive learning. Live right now in Sim!

Emir Karabeg

102,755 просмотров • 1 месяц назад

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot
0:31

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

Jiafei Duan✈️CoRL2026

38,957 просмотров • 3 лет назад