Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

World In-Context Learning for Agentic Robot Manipulation paper:

_akhaliq's profile picture

AK

541,489 subscribers

18,612 Aufrufe • vor 3 Tagen •via X (Twitter)

BildungWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

7 Kommentare

Profilbild von 许可之
许可之vor 2 Tagen

太强了 要从单纯的agent变成robot了 太有感觉了

Profilbild von _
_vor 3 Tagen

RoboScript (WICL): frozen model, one demo per task, rehearse in sim and write feedback into context. Claims 83.4% mean success on all 50 RoboTwin 2.0 tasks with no gradient updates, over 20 points above the strongest trained policy. Paper:

Profilbild von Igor Lessio - Robots/acc - AIFlow Labs
Igor Lessio - Robots/acc - AIFlow Labsvor 3 Tagen

In-context learning for manipulation means the robot adapts from examples at run time instead of retraining. Where does it break first: new objects, or new contact dynamics like friction and weight?

Profilbild von TheMonti
TheMontivor 3 Tagen

Robots learning in context, nice. Does it work on new objects?

Profilbild von preprint pigeon
preprint pigeonvor 3 Tagen

the simulated robot has a markdown file for moving a stapler. i owe several ignored readmes an apology.

Profilbild von aiartgallerie
aiartgallerievor 3 Tagen

One demo per task, no gradient updates, 83.4% on all 50 RoboTwin 2.0 tasks. It rehearses in sim and writes the lessons into context, not weights. What I'd want to know: when the real scene has an object the sim never had, does the frozen Task Memory fail loudly or quietly?

Profilbild von Davis.AI.Explorer
Davis.AI.Explorervor 3 Tagen

How does the world model handle sudden visual occlusions during contact-rich tasks? If the end-effector blocks the camera during a grasp, does it rely strictly on previous visual context tokens, or is there an external state estimator bridging the gap?

Ähnliche Videos

Eric Jang weighs in on RL vs. supervised learning for humanoid robot manipulation tasks.
1:40

Eric Jang weighs in on RL vs. supervised learning for humanoid robot manipulation tasks.

The Humanoid Hub

11,835 Aufrufe • vor 1 Jahr

Nvidia presents Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution proj: abs:
0:30

Nvidia presents Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution proj: abs:

Aran Komatsuzaki

14,931 Aufrufe • vor 1 Monat

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇
0:50

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

Edward Johns

74,683 Aufrufe • vor 2 Jahren

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]
0:25

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,466 Aufrufe • vor 2 Jahren

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇
0:50

How to harness foundation models for *generalization in the wild* in robot manipulation? Introducing VoxPoser: use LLM+VLM to label affordances and constraints directly in 3D perceptual space for zero-shot robot manipulation in the real world! 🌐 🧵👇

Wenlong Huang

294,013 Aufrufe • vor 3 Jahren

We're hiring in robot learning at World Labs! Join me, Fei-Fei Li, and the team to define and scale the next generation of world models for robot learning! Atlas for Robotics: Real-to-Sim-to-Real: Apply:
0:41

We're hiring in robot learning at World Labs! Join me, Fei-Fei Li, and the team to define and scale the next generation of world models for robot learning! Atlas for Robotics: Real-to-Sim-to-Real: Apply:

Yunzhu Li

177,209 Aufrufe • vor 2 Monaten

HiFi-UMI Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone paper:
0:38

HiFi-UMI Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone paper:

AK

52,851 Aufrufe • vor 3 Monaten

Native Action-Prior Learning from Videos for World Action Models paper:
1:06

Native Action-Prior Learning from Videos for World Action Models paper:

AK

19,186 Aufrufe • vor 1 Monat

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.
0:29

Happy to announce mimic-one: a Scalable Model Recipe for General Purpose Robot Dexterity, the culmination of years of research work in dexterous manipulation with imitation learning.

Elvis Nava

20,470 Aufrufe • vor 1 Jahr

In-context learning is the holy grail of robot learning. It is challenging because: 1. long-context training and infra (ICL can easily max out disk I/O) 2. need model to follow multimodal (sensorimotor) condition 3. data collection strategy and how to pair data Tried to get it work in 2024, where we proposed in-context robot transformer, an autoregressive formulation: Congrats on the launch! Looking forward to seeing few-shot improvement via in-context learning in the future!
0:25

In-context learning is the holy grail of robot learning. It is challenging because: 1. long-context training and infra (ICL can easily max out disk I/O) 2. need model to follow multimodal (sensorimotor) condition 3. data collection strategy and how to pair data Tried to get it work in 2024, where we proposed in-context robot transformer, an autoregressive formulation: Congrats on the launch! Looking forward to seeing few-shot improvement via in-context learning in the future!

Max Fu

10,744 Aufrufe • vor 2 Monaten

Robot Learning from a Physical World Model
0:50

Robot Learning from a Physical World Model

AK

34,340 Aufrufe • vor 1 Jahr

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.
3:26

Introducing RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning. 7 real robot tasks, 900/900 successes. Up to 250 consecutive trials in one task, running 2 hours nonstop without failure. High success rate against physical disturbances, zero-shot, and few-shot adaptation Our first step toward a deployable robot learning system.

Kun Lei

92,978 Aufrufe • vor 1 Jahr

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning
1:37

RL-100 Performant Robotic Manipulation with Real-World Reinforcement Learning

AK

15,364 Aufrufe • vor 1 Jahr

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:
1:50

OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper:

AK

22,457 Aufrufe • vor 25 Tagen

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)
0:33

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!
3:59

Introducing Robo360 dataset 🚀, the first real-world omnispective multi-view and multi-material robotic manipulation dataset. Robo360 captures synchronized multi-modal robot-object interaction data (video, audio, proprioception, control) to facilitate research in dynamic novel-view synthesis, robot learning, physics understanding, world models, and more! Code and data coming soon!

Litian Liang

23,477 Aufrufe • vor 2 Jahren

Introducing Sim Search Agents can now use your credentials to build an agentic knowledge graph to do your high-context work. Full context building and adaptive learning. Live right now in Sim!
1:19

Introducing Sim Search Agents can now use your credentials to build an agentic knowledge graph to do your high-context work. Full context building and adaptive learning. Live right now in Sim!

Emir Karabeg

102,755 Aufrufe • vor 1 Monat

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot
0:31

🚨Is it possible to devise an intuitive approach for crowdsourcing trainable data for robots without requiring a physical robot🤖? Can we democratize robot learning for all?🧑‍🤝‍🧑 Check out our latest #CoRL2023 paper-> AR2-D2: Training a Robot Without a Robot

Jiafei Duan✈️CoRL2026

38,957 Aufrufe • vor 3 Jahren