Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Excited to release RT-Affordance! We propose conditioning policies on visual affordance plans as an intermediate representation that allows us to learn new tasks without collecting any new robot trajectories. Website and paper: Here’s a short 🧵

Soroush Nasiriany

1,606 subscribers

27,495 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

7 Kommentare

Profilbild von Soroush Nasiriany

Soroush Nasirianyvor 1 Jahr

We want to make a robot’s job easy by telling it not only what to do but how to do it. Conditioning on language, goal images, and trajectory sketches are helpful, but they present their own challenges. Visual affordance plans are expressive and easy to specify!

Profilbild von Soroush Nasiriany

Soroush Nasirianyvor 1 Jahr

Our hierarchical model first predicts an affordance plan and then conditions the policy on the affordance plan. We co-train the model on web datasets (largest data source), robot trajectories, and a modest number of cheap-to-collect images labeled with affordances.

Profilbild von Soroush Nasiriany

Soroush Nasirianyvor 1 Jahr

Here’s the big kicker: we can adapt to new tasks and objects by just providing cheap-to-collect example images and annotating them with affordances. No additional costly robot demonstrations or teleoperation required!

Profilbild von Soroush Nasiriany

Soroush Nasirianyvor 1 Jahr

Please see the paper for more details. This was my internship project at Google DeepMind. A huge thank you to my awesome mentor @xiao_ted for supporting me and all of my lovely collaborators @SeanKirmani @TianliDing @smithlaura1028 @yukez @DannyDriess @DorsaSadigh!

Profilbild von Julien

Julienvor 1 Jahr

Did you guys reversed the models names ?

Profilbild von Soroush Nasiriany

Soroush Nasirianyvor 1 Jahr

thanks for pointing this out, fixed!

Profilbild von j15

j15vor 1 Jahr

Podcast about it:

Ähnliche Videos

How to scale visual affordance learning that is fine-grained, task-conditioned, works in-the-wild, in dynamic envs? Introducing Unsupervised Affordance Distillation (UAD): distills affordances from off-the-shelf foundation models, *all without manual labels*. Very excited this is nominated as Best Paper Finalist at #ICRA2025! 🧵👇

How to scale visual affordance learning that is fine-grained, task-conditioned, works in-the-wild, in dynamic envs? Introducing Unsupervised Affordance Distillation (UAD): distills affordances from off-the-shelf foundation models, all without manual labels. Very excited this is nominated as Best Paper Finalist at #ICRA2025! 🧵👇

Wenlong Huang

93,616 Aufrufe • vor 1 Jahr

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Super excited for the release of Robot Utility Models (RUMs)! RUMs is a simple method to build zero-shot robot policies that can solve useful tasks in completely new homes without any additional training often at 90%+ success rate. 🧵👇

Lerrel Pinto

56,591 Aufrufe • vor 1 Jahr

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

We introduce a system for fine-grained robotic manipulation! 🤖 What’s new? * We can control cheap robots to do surprisingly dexterous tasks * New technique that allows robots to learn fine motor skills A short thread 🧵

Chelsea Finn

264,399 Aufrufe • vor 3 Jahren

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Imagine robots learning new skills—without any robot data. Today, we're excited to release EgoZero: our first steps in training robot policies that operate in unseen environments, solely from data collected through humans wearing Aria smart glasses. 🧵👇

Lerrel Pinto

42,576 Aufrufe • vor 1 Jahr

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Vision-language models perform diverse tasks via in-context learning. Time for robots to do the same! Introducing In-Context Robot Transformer (ICRT): a robot policy that learns new tasks by prompting with robot trajectories, without any fine-tuning. [1/N]

Max Fu

40,447 Aufrufe • vor 1 Jahr

Excited to release FAST, our new robot action tokenizer! 🤖 Some highlights: - Simple autoregressive VLAs match diffusion VLA performance - Trains up to 5x faster - Works on all robot datasets we tested - First VLAs that work out-of-the-box in new environments! 🧵/

Excited to release FAST, our new robot action tokenizer! 🤖 Some highlights: - Simple autoregressive VLAs match diffusion VLA performance - Trains up to 5x faster - Works on all robot datasets we tested - First VLAs that work out-of-the-box in new environments! 🧵/

Karl Pertsch

90,625 Aufrufe • vor 1 Jahr

Meet Dobb·E: a home robot system that needs just 5 minutes of human teaching to learn new tasks. Dobb·E has visited 10 homes, learned 100+ tasks, and we are just getting started! Dobb·E is fully open-sourced (including hardware, models, and software): 🧵

Meet Dobb·E: a home robot system that needs just 5 minutes of human teaching to learn new tasks. Dobb·E has visited 10 homes, learned 100+ tasks, and we are just getting started! Dobb·E is fully open-sourced (including hardware, models, and software): 🧵

Lerrel Pinto

211,713 Aufrufe • vor 2 Jahren

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Policies trained on real robot data via imitation can be surprisingly capable. But for domains like dexterous manipulation, they are often not quite good enough: they move slowly, miss grasps, make unreliable contact, and fail under small perturbations. Can we improve them without any additional data collection on the real robot? In SCORE, we show that we can improve real-world diffusion/flow policies cheaply by using simulation to simply learn how to steer them on deployment. This leads to large gains in real-world success and speed across a variety of tasks, without requiring additional real-world experience: 🧵 (1/10)

Abhishek Gupta

33,455 Aufrufe • vor 14 Tagen

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

What if robots could learn real-world tasks from your perspective… without ever touching a robot? This is a system that trains robot policies using nothing but human-first, egocentric video data from smart glasses. No robots, no teleop, no sensors, just humans doing real tasks in the real world. Why it matters ✅ Learns robot policies from 20 minutes of human video; zero robot demos ✅ Generalizes to new objects, views, and even robot morphologies ✅ Uses 3D points for interpretable, spatially grounded learning ✅ Deploys directly to real-world robots with strong zero-shot success Thank you, Vincent Liu, for sharing!!! Learn more here: 🔗 Paper: 🌐 Website: 📍 BOOKMARK FOR LATER

Ilir Aliu - eu/acc

10,509 Aufrufe • vor 1 Jahr

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically *retrieved* “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are *not* collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant *sub-trajectories* from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

In my experience, robot 'generalists' are often jacks of all trades but masters of none. In training across multiple tasks and environments, robot policies fail to generalize robustly and effectively to each particular test setting. What if at test time, we non-parametrically retrieved “relevant” data from the training set and used it to significantly improve the performance of few-shot imitation learning to be robust to various test time scenes. Notably, we are not collecting lots of new data, just training more on sub-components of the same training data! Now, we’re certainly not the first to suggest retrieval, but in our new work - STRAP, we show how retrieving relevant sub-trajectories from offline datasets can significantly increase data reuse across tasks, when paired with an appropriate metric space. A 🧵 (1/7)

Abhishek Gupta

12,045 Aufrufe • vor 1 Jahr

How can a single cell learn without a brain? We explore this in my new paper with Wallace Marshall! We discovered that single cells may learn using molecules similar to those that animal brains use to learn, like CaMKII. Cells can also propagate memory states to their progeny! 🧵1/n

How can a single cell learn without a brain? We explore this in my new paper with Wallace Marshall! We discovered that single cells may learn using molecules similar to those that animal brains use to learn, like CaMKII. Cells can also propagate memory states to their progeny! 🧵1/n

Deepa Rajan, PhD

75,700 Aufrufe • vor 2 Monaten

We are launching a new feature on Poe today: user-created bots. This initially allows anyone to create a new bot based on a short text prompt and an existing bot used as a base. We currently support Claude instant and ChatGPT as base bots.

We are launching a new feature on Poe today: user-created bots. This initially allows anyone to create a new bot based on a short text prompt and an existing bot used as a base. We currently support Claude instant and ChatGPT as base bots.

Adam D'Angelo

388,247 Aufrufe • vor 3 Jahren

Excited to share our new paper! “Forecasting Downstream Performance of LLMs With Proxy Metrics” w/ my amazing advisors Siva Reddy, Marius Mosbach, 🇺🇦 Dzmitry Bahdanau Cross-entropy loss is a poor predictor of how models perform on downstream tasks (esp. reasoning). We propose something better: proxy metrics computed over expert reasoning traces. 🧵 Thread below 👇

Excited to share our new paper! “Forecasting Downstream Performance of LLMs With Proxy Metrics” w/ my amazing advisors Siva Reddy, Marius Mosbach, 🇺🇦 Dzmitry Bahdanau Cross-entropy loss is a poor predictor of how models perform on downstream tasks (esp. reasoning). We propose something better: proxy metrics computed over expert reasoning traces. 🧵 Thread below 👇

Arkil Patel

99,057 Aufrufe • vor 1 Monat

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy? 🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️ ✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement Let's squeeze every bit of signal out of human data! 🌐 Website: 📄 Paper: 💻 Code: 📹 Video: 🧵 1/n

Zhi (Leo) Wang

109,407 Aufrufe • vor 1 Monat

🔥Convolutional Kolmogorov-Arnold Networks Preprint out now🔥 I'm very excited to release our preprint on Arxiv. Also, my first paper. We proposed a novel architecture that shows potential to compete with CNNs. Want to learn more about it? 🧵Follow this thread👇

🔥Convolutional Kolmogorov-Arnold Networks Preprint out now🔥 I'm very excited to release our preprint on Arxiv. Also, my first paper. We proposed a novel architecture that shows potential to compete with CNNs. Want to learn more about it? 🧵Follow this thread👇

Alex Bodner

102,787 Aufrufe • vor 2 Jahren

Super excited to share the last paper of my PhD: "Hallucination in World Models is Predictable and Preventable"✨ We train a 350M-param generative world model on a large dataset w/ 210 tasks and show that we can predict *when* hallucination happens and use that to fix it! 🧵1/n

Super excited to share the last paper of my PhD: "Hallucination in World Models is Predictable and Preventable"✨ We train a 350M-param generative world model on a large dataset w/ 210 tasks and show that we can predict when hallucination happens and use that to fix it! 🧵1/n

Nicklas Hansen

53,264 Aufrufe • vor 20 Tagen

Introducing Vulcan - a new Al robot controller This is an important safety feature that allows F.03 to lose up to 3 actuators in the lower body and still hobble around without falling Check it out

Introducing Vulcan - a new Al robot controller This is an important safety feature that allows F.03 to lose up to 3 actuators in the lower body and still hobble around without falling Check it out

Brett Adcock

355,634 Aufrufe • vor 3 Monaten

🚨 New Paper Training an LLM to speak low-resource language (EACL workshop, 2026) Tulu is spoken by 2M+ people in coastal Karnataka and LLMs basically can't speak it. We got to 85% grammar accuracy without fine-tuning anything or collecting a single new training example.

🚨 New Paper Training an LLM to speak low-resource language (EACL workshop, 2026) Tulu is spoken by 2M+ people in coastal Karnataka and LLMs basically can't speak it. We got to 85% grammar accuracy without fine-tuning anything or collecting a single new training example.

Lossfunk

120,884 Aufrufe • vor 4 Monaten

Turns out that vision-language models can control robots too. The secret is to just finetune them to print out the actions (literally, as text). Really excited about our new result, the successor to RT-1. RT-2 is a pre-trained VLM: Short 🧵👇

Turns out that vision-language models can control robots too. The secret is to just finetune them to print out the actions (literally, as text). Really excited about our new result, the successor to RT-1. RT-2 is a pre-trained VLM: Short 🧵👇

Sergey Levine

165,052 Aufrufe • vor 3 Jahren

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

Mandi Zhao

120,892 Aufrufe • vor 1 Jahr