Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

We can teach LLMs to write better robot code through natural language feedback. But can LLMs remember what they were taught and improve their teachability over time? Introducing our latest work, Learning to Learn Faster from Human Feedback with Language Model Predictive Control

jackyliang42's profile picture

Jacky Liang

7,434 subscribers

86,652 görüntüleme • 2 yıl önce •via X (Twitter)

EğitimBilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

13 Yorum

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

TL;DR We improve LLMs’ in-context teachability by fine-tuning them to model human-robot interactions. Here’s an example of robot teaching sessions before and after finetuning. Before, the LLM requires many corrections to do a high-five; after, the model gets there much faster

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

Given a dataset of these chat-based teaching sessions, how do we improve LLMs’ teachability (number of chat turns it takes to reach task success)? We want to improve teachability not just on train tasks, but also on test tasks and test embodiments.

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

Instead of finetuning to directly output final answers, we propose Language Model Predictive Control: 1) Finetune the LLM to predict the entire chat session 2) During inference: sample rollouts of future chat sessions, return the first response of the shortest successful chat

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

Compared to retrieval (RAG) and directly predict the answer (LMPC-Skip), our method (LMPC-Rollouts) achieves the highest improvement in teachability over the base model (PaLM 2-S) With 1 chat turn, LMPC-Skip does the best, but LMPC-Rollouts improves more with corrective feedback

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

On test tasks, LMPC-Rollouts have the highest success rate when the chat has 2+ turns; it also has the highest good rating rate (if human teachers rated each chat response positively).

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

While we only fine-tune on 3 embodiments (robot dog, mobile manipulator, aloha), we also see improvements on test embodiments (bimanual kuka, kuka+hand), both have different robot APIs and tasks

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

Our method enables better teaching on real robots too! Here’s an example with the robot dog…

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

…and here’s another example for our mobile manipulation robot

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

See more details in our paper, and videos and demo on our website:

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

This was a huge collaboration from many folks at Google DeepMind Robotics. I learned a ton from everyone and am super grateful for the amazing teamwork! Special thanks to @xf1280 @Stacormed @andyzeng_! Big shoutouts to our incredible collaborators: @montseglz @JMarakiii …

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

… @bauzavillalonga Matthew Bennice @AlexBewleyAI @AdilDostmohamed @ChuyuanFu @NimTheCoder Marissa Giustina @keerthanpg @lqh20 Jan Humplik, Jasmine Hsu, Nikhil Joshi, Ben Jyenis, Chase Kew, @SeanKirmani Edward Lee, @kuanghueilee, Assaf Hurwitz Michaely, Joss Moore, Ken Oslund, …

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

…, Dushyant Rao, @allenzren Baruch Tabanpour @QuanVng @ayzwah @xiao_ted Ying Xu, Vincent Zhuang, as well as our incredible advising leads: Peng Xu, Erik Frey, Ken Caluwaerts, Tingnan Zhang, @brian_ichter @JonathanTompson @leilatakayama Vincent Vanhoucke @IzhakShafran …

Jacky Liang profil fotoğrafı
Jacky Liang2 yıl önce

…Maja Mataric @DorsaSadigh Nicolas Heess @Kanishka_Rao Nik Stewart, Jie Tan, Carolina Parada Thanks everyone!!

Benzer Videolar

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).
3:10

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Physical Intelligence

116,845 görüntüleme • 1 yıl önce

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!
1:29

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,773 görüntüleme • 1 yıl önce

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:
1:03

Robot AI brains, aka Vision-Language-Action models, cannot adapt to new tasks as easily as LLMs like Gemini, ChatGPT, or Grok. LLMs can adapt quickly with their in-context learning (ICL) capabilities. But can we inject ICL abilities into a pre-trained VLA like pi0? Yes! Introducing RICL (Retraining for In-Context Learning), our Conference on Robot Learning (CoRL) 2025 paper. Our RICL-pi0 model can adapt to unseen objects, novel motions, and new scenes with just ICL and RAG (retrieval-augmented generation). RICL-pi0 also boosts performance on the long-tail of tasks. A quick 1 minute video summary:

Kaustubh Sridhar

52,158 görüntüleme • 11 ay önce

Yann LeCun says we're fooled by LLMs because they manipulate language well, and we associate that with intelligence But language fluency doesn't mean underlying intelligence Every generation since the 1950s claimed its technique was the ticket to human-level AI All were wrong. "this generation with LLMs is also wrong"
1:35

Yann LeCun says we're fooled by LLMs because they manipulate language well, and we associate that with intelligence But language fluency doesn't mean underlying intelligence Every generation since the 1950s claimed its technique was the ticket to human-level AI All were wrong. "this generation with LLMs is also wrong"

Haider.

625,577 görüntüleme • 7 ay önce

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:
2:58

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w Stanford University and UC Berkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables long-horizon, dexterous manipulation tasks like preparing trail-mix, packing a ziploc bag, and cleaning dishes:

Lucy Shi

122,827 görüntüleme • 2 yıl önce

Geoffrey Hinton says people still think LLMs are different from us, but they’re not LLMs don’t just generate words; they reflect how we process meaning: "they’re our best model of how language works" old linguistic theories failed to explain language; neural nets using feature vectors finally do
0:53

Geoffrey Hinton says people still think LLMs are different from us, but they’re not LLMs don’t just generate words; they reflect how we process meaning: "they’re our best model of how language works" old linguistic theories failed to explain language; neural nets using feature vectors finally do

Haider.

193,258 görüntüleme • 1 yıl önce

Demis Hassabis says large language models (LLMs) can search and plan but they forget everything once the session ends. This is because today’s models don’t truly “learn” after deployment. He sees continual learning and better memory as key missing ingredients for AGI. Efficient memory, like the brain’s ability to store only what matters, will be crucial. LLMs are powerful, but without memory, they have a "goldfish brain."
4:40

Demis Hassabis says large language models (LLMs) can search and plan but they forget everything once the session ends. This is because today’s models don’t truly “learn” after deployment. He sees continual learning and better memory as key missing ingredients for AGI. Efficient memory, like the brain’s ability to store only what matters, will be crucial. LLMs are powerful, but without memory, they have a "goldfish brain."

Wes Roth

38,833 görüntüleme • 5 ay önce

Yann LeCun says LLMs are strongest in domains where language itself is the substrate of reasoning, like math and code They can solve problems, prove theorems, and write programs — but they are not creative mathematicians, software architects, or computer scientists "their role is to help humans build"
1:28

Yann LeCun says LLMs are strongest in domains where language itself is the substrate of reasoning, like math and code They can solve problems, prove theorems, and write programs — but they are not creative mathematicians, software architects, or computer scientists "their role is to help humans build"

Haider.

347,295 görüntüleme • 2 ay önce

Francois Chollet says LLMs aren't enough for human-like continual learning They store skills as vector programs learned via gradient descent — not efficient, not adaptive True AGI needs to learn from experience and generalize fast "LLMs can be part of AGI, but not the solution"
1:06

Francois Chollet says LLMs aren't enough for human-like continual learning They store skills as vector programs learned via gradient descent — not efficient, not adaptive True AGI needs to learn from experience and generalize fast "LLMs can be part of AGI, but not the solution"

Haider.

50,251 görüntüleme • 8 ay önce

NEWS: Humanoid robot startup Figure announced Helix today, their "in-house AI that reasons like a human." "Our robots equipped with Helix can now pick up virtually any household object without any code or prior training. Helix uses a single set of neural network weights to learn all behaviors." "We're introducing Helix, a generalist Vision-Language-Action (VLA) model that unifies perception, language understanding, and learned control to overcome multiple longstanding challenges in robotics."
2:15

NEWS: Humanoid robot startup Figure announced Helix today, their "in-house AI that reasons like a human." "Our robots equipped with Helix can now pick up virtually any household object without any code or prior training. Helix uses a single set of neural network weights to learn all behaviors." "We're introducing Helix, a generalist Vision-Language-Action (VLA) model that unifies perception, language understanding, and learned control to overcome multiple longstanding challenges in robotics."

Sawyer Merritt

181,177 görüntüleme • 1 yıl önce

Many AI-savvy programmers are now coding very differently than before, by using LLMs to help with their work. You’ll learn these emerging best practices in “Pair Programming with a Large Language Model,” by Google's Laurence Moroney 🇺🇸🇮🇪 🏴󠁧󠁢󠁷󠁬󠁳󠁿. This short course covers using LLMs to simplify and improve your code, assist with debugging, and minimize technical debt by having AI document your code. The use of LLMs as a programming companion is an important shift that's well worth every developer staying on top of. Please check this out!
1:04

Many AI-savvy programmers are now coding very differently than before, by using LLMs to help with their work. You’ll learn these emerging best practices in “Pair Programming with a Large Language Model,” by Google's Laurence Moroney 🇺🇸🇮🇪 🏴󠁧󠁢󠁷󠁬󠁳󠁿. This short course covers using LLMs to simplify and improve your code, assist with debugging, and minimize technical debt by having AI document your code. The use of LLMs as a programming companion is an important shift that's well worth every developer staying on top of. Please check this out!

Andrew Ng

373,486 görüntüleme • 2 yıl önce

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.
0:50

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,708 görüntüleme • 3 yıl önce

🚀 Introducing Cover-Agent 🧪 An open-source tool that includes a reimplementation of Meta's TestGen-LLM for automatically enhancing test suites. Manager: "We must improve old test suites for better code coverage. Can you handle it?" Me: "Sure, my favorite task... (Not!) 🤷‍♂️" Meta's team had the idea of using LLMs to enhance test suites—sounds great, right? But they didn't release their code We did. We'd love your feedback as we work on improving this baseline In this video, I'll review TestGen-LLM, share insights, and introduce you to Cover-Agent
6:11

🚀 Introducing Cover-Agent 🧪 An open-source tool that includes a reimplementation of Meta's TestGen-LLM for automatically enhancing test suites. Manager: "We must improve old test suites for better code coverage. Can you handle it?" Me: "Sure, my favorite task... (Not!) 🤷‍♂️" Meta's team had the idea of using LLMs to enhance test suites—sounds great, right? But they didn't release their code We did. We'd love your feedback as we work on improving this baseline In this video, I'll review TestGen-LLM, share insights, and introduce you to Cover-Agent

Itamar Friedman

139,228 görüntüleme • 2 yıl önce

New #ACL2025NLP Paper! 🎉 Curious what AI thinks about YOU? We interact with AI every day, offering all kinds of feedback, both implicit ✏️ and explicit 👍. What if we used this feedback to personalize your AI assistant to you? Introducing SynthesizeMe! An approach for creating natural language personal user models from your interactions. 🧵
1:10

New #ACL2025NLP Paper! 🎉 Curious what AI thinks about YOU? We interact with AI every day, offering all kinds of feedback, both implicit ✏️ and explicit 👍. What if we used this feedback to personalize your AI assistant to you? Introducing SynthesizeMe! An approach for creating natural language personal user models from your interactions. 🧵

Michael Ryan

47,080 görüntüleme • 1 yıl önce

New Anthropic research: Emotion concepts and their function in a large language model. All LLMs sometimes act like they have emotions. But why? We found internal representations of emotion concepts that can drive Claude’s behavior, sometimes in surprising ways.
4:52

New Anthropic research: Emotion concepts and their function in a large language model. All LLMs sometimes act like they have emotions. But why? We found internal representations of emotion concepts that can drive Claude’s behavior, sometimes in surprising ways.

Anthropic

3,921,366 görüntüleme • 3 ay önce

Humans use pointing to communicate plans intuitively. Compared to language, pointing gives more precise guidance to robot behaviors. Can we teach a robot how to point like humans? Introducing RoboPoint 🤖👉, an open-source VLM instruction-tuned to point. Check out our new work:
1:24

Humans use pointing to communicate plans intuitively. Compared to language, pointing gives more precise guidance to robot behaviors. Can we teach a robot how to point like humans? Introducing RoboPoint 🤖👉, an open-source VLM instruction-tuned to point. Check out our new work:

Jiafei Duan

64,377 görüntüleme • 2 yıl önce

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:
2:41

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,457 görüntüleme • 1 yıl önce

DocETL is a system we’ve been building at Berkeley for the past two years to make large-scale unstructured data analysis reliable and efficient. It powers our broader stack—used by journalists, public defenders, and researchers—to extract, transform, and reason over messy documents with LLMs. As part of making the DocETL ecosystem easier to use, we’re introducing a natural language–to–pipeline generator! Our hosted version is free to use BUT we're collecting the data so we can build better tools.
2:03

DocETL is a system we’ve been building at Berkeley for the past two years to make large-scale unstructured data analysis reliable and efficient. It powers our broader stack—used by journalists, public defenders, and researchers—to extract, transform, and reason over messy documents with LLMs. As part of making the DocETL ecosystem easier to use, we’re introducing a natural language–to–pipeline generator! Our hosted version is free to use BUT we're collecting the data so we can build better tools.

Shreya Shankar

14,428 görüntüleme • 8 ay önce