正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

World In-Context Learning for Agentic Robot Manipulation paper:

AK

541,489 subscribers

18,612 次观看 • 3 天前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

7 条评论

许可之的头像

许可之2 天前

太强了要从单纯的agent变成robot了太有感觉了

_ 的头像

_3 天前

RoboScript (WICL): frozen model, one demo per task, rehearse in sim and write feedback into context. Claims 83.4% mean success on all 50 RoboTwin 2.0 tasks with no gradient updates, over 20 points above the strongest trained policy. Paper:

Igor Lessio - Robots/acc - AIFlow Labs 的头像

Igor Lessio - Robots/acc - AIFlow Labs3 天前

In-context learning for manipulation means the robot adapts from examples at run time instead of retraining. Where does it break first: new objects, or new contact dynamics like friction and weight?

TheMonti 的头像

TheMonti3 天前

Robots learning in context, nice. Does it work on new objects?

preprint pigeon 的头像

preprint pigeon3 天前

the simulated robot has a markdown file for moving a stapler. i owe several ignored readmes an apology.

aiartgallerie 的头像

aiartgallerie3 天前

One demo per task, no gradient updates, 83.4% on all 50 RoboTwin 2.0 tasks. It rehearses in sim and writes the lessons into context, not weights. What I'd want to know: when the real scene has an object the sim never had, does the frozen Task Memory fail loudly or quietly?

Davis.AI.Explorer 的头像

Davis.AI.Explorer3 天前

How does the world model handle sudden visual occlusions during contact-rich tasks? If the end-effector blocks the camera during a grasp, does it rely strictly on previous visual context tokens, or is there an external state estimator bridging the gap?

相关视频

Eric Jang weighs in on RL vs. supervised learning for humanoid robot manipulation tasks.

Eric Jang weighs in on RL vs. supervised learning for humanoid robot manipulation tasks.

The Humanoid Hub

11,835 次观看 • 1 年前

Nvidia presents Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution proj: abs:

Nvidia presents Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution proj: abs:

Aran Komatsuzaki

14,931 次观看 • 1 个月前

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

Edward Johns

74,683 次观看 • 2 年前

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,466 次观看 • 2 年前

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 次观看 • 1 年前

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

How to harness foundation models for generalization in the wild in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

Wenlong Huang

294,013 次观看 • 3 年前

We're hiring in robot learning at World Labs! Join me, Fei-Fei Li, and the team to define and scale the next generation of world models for robot learning! Atlas for Robotics: Real-to-Sim-to-Real: Apply:

We're hiring in robot learning at World Labs! Join me, Fei-Fei Li, and the team to define and scale the next generation of world models for robot learning! Atlas for Robotics: Real-to-Sim-to-Real: Apply:

Yunzhu Li

177,209 次观看 • 2 个月前

HiFi-UMI Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone paper:

HiFi-UMI Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone paper:

AK

52,851 次观看 • 3 个月前

Native Action-Prior Learning from Videos for World Action Models paper:

Native Action-Prior Learning from Videos for World Action Models paper:

AK

19,186 次观看 • 1 个月前

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Elvis Nava

20,470 次观看 • 1 年前

In-context learning is the holy grail of robot learning. It is challenging because: 1. long-context training and infra (ICL can easily max out disk I/O) 2. need model to follow multimodal (sensorimotor) condition 3. data collection strategy and how to pair data Tried to get it work in 2024, where we proposed in-context robot transformer, an autoregressive formulation: Congrats on the launch! Looking forward to seeing few-shot improvement via in-context learning in the future!

In-context learning is the holy grail of robot learning. It is challenging because: 1. long-context training and infra (ICL can easily max out disk I/O) 2. need model to follow multimodal (sensorimotor) condition 3. data collection strategy and how to pair data Tried to get it work in 2024, where we proposed in-context robot transformer, an autoregressive formulation: Congrats on the launch! Looking forward to seeing few-shot improvement via in-context learning in the future!

Max Fu

10,744 次观看 • 2 个月前

Robot Learning from a Physical World Model

Robot Learning from a Physical World Model

AK

34,340 次观看 • 1 年前

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

92,978 次观看 • 1 年前

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

AK

15,364 次观看 • 1 年前

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:

AK

22,457 次观看 • 25 天前

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 次观看 • 1 年前

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Litian Liang

23,477 次观看 • 2 年前

Introducing Sim Search Agents can now use your credentials to build an agentic knowledge graph to do your high-context work. Full context building and adaptive learning. Live right now in Sim!

Introducing Sim Search Agents can now use your credentials to build an agentic knowledge graph to do your high-context work. Full context building and adaptive learning. Live right now in Sim!

Emir Karabeg

102,755 次观看 • 1 个月前

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

Jiafei Duan✈️CoRL2026

38,957 次观看 • 3 年前