Loading video...

Video Failed to Load

Go Home

Reinforcement Learning from Human Feedback (RLHF) is gaining traction. This field aims to make AI more responsible by including human values and preferences. In this video, Nathan Lambert, a research scientist and RLHF team lead at Hugging Face explores its inner workings, applications and industry impact. RLHF has gained...

muratcan's profile picture

Muratcan Koylan

22,145 subscribers

27,168 views • 3 years ago •via X (Twitter)

Science & TechnologyEducationHealth & Wellness
LIVE

Anya Rossi• Live Now

Private livecam show

8 Comments

Muratcan Koylan's profile picture
Muratcan Koylan3 years ago

@huggingface You can watch the entire video here. [Invited talk by Nathan Lambert on March 9, 2023 at UCL DARK.]

Muratcan Koylan's profile picture
Muratcan Koylan3 years ago

If you are interested in prompt engineering and LLM models, I highly recommend:

Nicolas's profile picture
Nicolas3 years ago

@natolambert @huggingface Thank you for sharing this insightful content. Is truly fascinating to see how AI is evolving to incorporate human values and preferences

Muratcan Koylan's profile picture
Muratcan Koylan3 years ago

@natolambert @huggingface Thank you for your support, Nicolas! Always a pleasure! My last rt indicates how RLHF can be automated by ai agents and some researchers claim that the results are better than human feedback 🫨

Nathan Lambert's profile picture
Nathan Lambert3 years ago

@huggingface Here are some much, much more recent talks covering RLHF. Thanks for sharing my work!

Muratcan Koylan's profile picture
Muratcan Koylan3 years ago

@huggingface Thanks for the contribution to the community, Nathan 🙏🏻

GAIO's profile picture
GAIO3 years ago

@natolambert @huggingface RLHF is similar to the psychology of Pavlov’s dog. but how do you incentivize / reward an AI? what are AI treats?

Muratcan Koylan's profile picture
Muratcan Koylan3 years ago

Pavlov uses food as a treat. AI models receives numerical rewards to adjust the internal weights and biases of the model to improve its performance. Think like "good" or "bad" outcomes. Human Feedback in RLHF, human reviewers rank them based on quality. With enough feedback, the model gets better at producing the desired outputs. Here you can find more details about the possibilities and limitations of RLHF:

Related Videos

New short course on Reinforcement Learning from Human Feedback! RLHF is one of the key techniques that led to the rise of modern LLMs. It is used to align LLMs with human preferences, to make them more honest, helpful and harmless, by (i) learning a reward function that mimics human preferences, as expressed in human-provided labels, then, (ii) tuning an LLM to generate outputs that receive a high reward. In this course, taught by Nikita Namjoshi, Developer Advocate for GenAI at Google Cloud, you'll learn the details of how RLHF works, including how to apply it to tune an LLM for your own applications. You'll also use an open source library to tune a base LLM to align with human preferences expressed in a training set, and evaluate the tuned model by comparing its responses before and after RLHF-tuning. Please sign up here!
3:26

New short course on Reinforcement Learning from Human Feedback! RLHF is one of the key techniques that led to the rise of modern LLMs. It is used to align LLMs with human preferences, to make them more honest, helpful and harmless, by (i) learning a reward function that mimics human preferences, as expressed in human-provided labels, then, (ii) tuning an LLM to generate outputs that receive a high reward. In this course, taught by Nikita Namjoshi, Developer Advocate for GenAI at Google Cloud, you'll learn the details of how RLHF works, including how to apply it to tune an LLM for your own applications. You'll also use an open source library to tune a base LLM to align with human preferences expressed in a training set, and evaluate the tuned model by comparing its responses before and after RLHF-tuning. Please sign up here!

Andrew Ng

205,605 views • 2 years ago

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, VP of AI at AMD. Available now at Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here:
2:27

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, VP of AI at AMD. Available now at Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here:

Andrew Ng

132,304 views • 11 months ago

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:
2:41

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,697 views • 1 year ago

Another friday afternoon talk in attempt of open science. This is a re-record of a talk I gave at two Chantham House Rules workshops this fall: The History and Risks of Reinforcement Learning and Human Feedback (yes, the and is clever and intentional). This tries to broaden the discussion around RLHF to include more stakeholders and diverse ideas about how to integrate or measure values (can you at all?!?). I'm trying to re-record every talk that I give that isn't to a public audience 🫡😅, good practice at least. Slides: Youtube: Paper:
17:23

Another friday afternoon talk in attempt of open science. This is a re-record of a talk I gave at two Chantham House Rules workshops this fall: The History and Risks of Reinforcement Learning and Human Feedback (yes, the and is clever and intentional). This tries to broaden the discussion around RLHF to include more stakeholders and diverse ideas about how to integrate or measure values (can you at all?!?). I'm trying to re-record every talk that I give that isn't to a public audience 🫡😅, good practice at least. Slides: Youtube: Paper:

Nathan Lambert

54,042 views • 2 years ago

BREAKING: AI has eaten the Internet. Data labeling is so over. & $30 trillion of human work is on the verge of automation. Inside The $2.2B AI Research Accelerator, Turing Founder & CEO, Jonathan Siddharth (), joins Sourcery to break down the severe power shift in AI training: from commodity data labeling → expert research Positioning Turing apart from AI data providers like Scale AI, Mercor, & Surge. (00:00) AI Ate The Internet (00:49) Training Superintelligence: the race to AGI (02:31) Viral tweet (03:24) What Turing actually does (04:43) The internet data is “used up” — where will new data come from? (05:34) Four pillars of superintelligence: multimodality, reasoning, tool use, coding (06:07) Automating $30T of global knowledge work (09:18) The $1B revenue opportunity (10:59) Why Turing is a research-first accelerator, not a data labeler (13:45) Jonathan’s Stanford AI Lab roots & founding DNA (17:57) How models are built: pre-training vs. post-training (20:14) RLHF, reinforcement learning, & “breaking the models” (25:19) GPT-5 and the myth of rapid takeoff (30:46) Safety debates and human-in-the-loop systems (34:53) Closing Enterprise Gap: finance, insurance, & pharma (39:23) Why proprietary enterprise data is the next moat in AI
49:29

BREAKING: AI has eaten the Internet. Data labeling is so over. & $30 trillion of human work is on the verge of automation. Inside The $2.2B AI Research Accelerator, Turing Founder & CEO, Jonathan Siddharth (), joins Sourcery to break down the severe power shift in AI training: from commodity data labeling → expert research Positioning Turing apart from AI data providers like Scale AI, Mercor, & Surge. (00:00) AI Ate The Internet (00:49) Training Superintelligence: the race to AGI (02:31) Viral tweet (03:24) What Turing actually does (04:43) The internet data is “used up” — where will new data come from? (05:34) Four pillars of superintelligence: multimodality, reasoning, tool use, coding (06:07) Automating $30T of global knowledge work (09:18) The $1B revenue opportunity (10:59) Why Turing is a research-first accelerator, not a data labeler (13:45) Jonathan’s Stanford AI Lab roots & founding DNA (17:57) How models are built: pre-training vs. post-training (20:14) RLHF, reinforcement learning, & “breaking the models” (25:19) GPT-5 and the myth of rapid takeoff (30:46) Safety debates and human-in-the-loop systems (34:53) Closing Enterprise Gap: finance, insurance, & pharma (39:23) Why proprietary enterprise data is the next moat in AI

Molly O’Shea

70,094 views • 11 months ago

Jev Founder, Diogo Almeida (ex-OpenAI): "The next era is not the Claude Code or Codex era, they are still part of the assistance era with human in the loop - JEV is what comes next for LLMs x200 faster, x400 cheaper, 0 hallucination, no human in the loop - that's JEV, this is how LLMs will look like" in 36-minute tech talk, Jev Founder explained why RLHF isn't a thing anymore and how modern LLMs will be built this talk is worth more than a Stanford Machine Learning degree watch today no matter what, then learn how to become a Jev Engineer in the article below
36:14

Jev Founder, Diogo Almeida (ex-OpenAI): "The next era is not the Claude Code or Codex era, they are still part of the assistance era with human in the loop - JEV is what comes next for LLMs x200 faster, x400 cheaper, 0 hallucination, no human in the loop - that's JEV, this is how LLMs will look like" in 36-minute tech talk, Jev Founder explained why RLHF isn't a thing anymore and how modern LLMs will be built this talk is worth more than a Stanford Machine Learning degree watch today no matter what, then learn how to become a Jev Engineer in the article below

Codez

445,940 views • 6 days ago

Revolutionizing Move Programming with OpenLedger In this demo, we showcase how Move datasets contributed by data providers to OpenLedger’s datanets are used to fine-tune specialized models with LoRA fine-tuning. As seen in the video, we showcase an example on how builders can deploy a Move-specialized model that powers Co-pilot agents using our no-code model fine-tuning platform. This is the future of AI and Web3 innovation. Watch this space to see more specialised models and data feeds being built for next generation agents on top of OpenLedger #Move
3:58

Revolutionizing Move Programming with OpenLedger In this demo, we showcase how Move datasets contributed by data providers to OpenLedger’s datanets are used to fine-tune specialized models with LoRA fine-tuning. As seen in the video, we showcase an example on how builders can deploy a Move-specialized model that powers Co-pilot agents using our no-code model fine-tuning platform. This is the future of AI and Web3 innovation. Watch this space to see more specialised models and data feeds being built for next generation agents on top of OpenLedger #Move

OpenLedger

61,697 views • 1 year ago

[RLHF] by Hand ✍️ Yesterday, Jan Leike (Jan Leike) announced he is joining #Anthropic to lead their "super-alignment" mission. He is the co-inventor of Reinforcement Learning with Human Feedback (#RLHF). How does RLHF work? [1] Given ↳ Reward Model (RM) ↳ Large Language Model (LLM) ↳ Two (Prompt, Next) Pairs 🟪 TRAIN RM Goal: Learn to give higher rewards to winners [2] Preferences ↳ A human reviews the two pairs and picks a "winner" ↳ (doc is, him) Embeddings ↳ This prompt has never received human feedback directly ↳ [S] is the special start symbol [11] Transformer ↳ Attention (yellow) ↳ Feed Forward (4x2 weight and bias matrix) ↳ Output: 3 "transformed" feature vector, one per position ↳ More details in my previous post 8. Transformer [] [12] Output Probabilities ↳ Apply a linear layer to map each transformed feature vector to a probability distribution over the vocabulary. [13] Sample ↳ Apply the greedy method, which is to pick the word with the highest score ↳ For output 1 and 2, the model accurately predicts the next word ↳ For 3rd output position, the model's predicts "him" [14] Reward Model ↳ The new pair (CEO is, him) is fed to the reward model ↳ The process is same as [3]-[6] ↳ Output: Reward = 3 [15] Loss Gradient ↳ We set the loss as the negative of the reward. ↳ The loss gradient is simply a constant -1. ↳ Run backpropagation and gradient descent to update LLM's weights and biases (red border)
1:01

[RLHF] by Hand ✍️ Yesterday, Jan Leike (Jan Leike) announced he is joining #Anthropic to lead their "super-alignment" mission. He is the co-inventor of Reinforcement Learning with Human Feedback (#RLHF). How does RLHF work? [1] Given ↳ Reward Model (RM) ↳ Large Language Model (LLM) ↳ Two (Prompt, Next) Pairs 🟪 TRAIN RM Goal: Learn to give higher rewards to winners [2] Preferences ↳ A human reviews the two pairs and picks a "winner" ↳ (doc is, him) Embeddings ↳ This prompt has never received human feedback directly ↳ [S] is the special start symbol [11] Transformer ↳ Attention (yellow) ↳ Feed Forward (4x2 weight and bias matrix) ↳ Output: 3 "transformed" feature vector, one per position ↳ More details in my previous post 8. Transformer [] [12] Output Probabilities ↳ Apply a linear layer to map each transformed feature vector to a probability distribution over the vocabulary. [13] Sample ↳ Apply the greedy method, which is to pick the word with the highest score ↳ For output 1 and 2, the model accurately predicts the next word ↳ For 3rd output position, the model's predicts "him" [14] Reward Model ↳ The new pair (CEO is, him) is fed to the reward model ↳ The process is same as [3]-[6] ↳ Output: Reward = 3 [15] Loss Gradient ↳ We set the loss as the negative of the reward. ↳ The loss gradient is simply a constant -1. ↳ Run backpropagation and gradient descent to update LLM's weights and biases (red border)

Tom Yeh

79,916 views • 2 years ago

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.
0:25

Karpathy's prediction about RL is coming true now! He called reward functions unreliable and argued that a single reward number is too low-dimensional to teach an agent what "good" means for complex tasks. To solve this, Agents need a knowledge-guided review as a higher-dimensional feedback channel. Every major AI lab trains models with RL today (OpenAI, Anthropic, DeepSeek). And their key bottleneck has always been the reward functions. GRPO by DeepSeek worked well for math and code because the environment gave a binary signal. But for real agent tasks, someone still has to hand-code the scoring function. That takes days and breaks every time the pipeline changes. RULER (implemented in OpenPipe ART, 10k stars) addresses the exact problem Karpathy identified. The reward criteria are defined in plain English, and an LLM evaluates each trajectory against that description to provide feedback for training. I trained a Qwen3 1.4B agent that plays 2048 using GRPO with this exact workflow. In this case, the agent saw the board, picked a direction, and RULER evaluated the outcome, all from this natural language definition. You can see the full implementation on GitHub and try it yourself. Here's the ART Repo: (don't forget to star it ⭐ ) Just like RLHF replaced manual rankings and GRPO replaced the critic model, natural language rewards are replacing hand-coded scoring functions. RL reward engineering is now prompt engineering. I wrote a full walkthrough covering RL for LLM agents, from RLHF to GRPO to RULER, in the article below.

Avi Chawla

350,798 views • 4 months ago

Important announcement (with job opportunities!): I’m thrilled to share that I just joined Lila Sciences as SVP of Open-Endedness! Lila is a new name in the AI space, but one you will be hearing a lot from. Their unique mission to pursue Scientific Superintelligence could not align better with my interest in open-ended creativity. Lila is about the entire scientific enterprise, not just a single constrained domain like drug discovery. And of course, the history of scientific progress is an unmistakable precedent for open-ended discovery and Why Greatness Cannot Be Planned. For AI itself to join this epic quest beyond just a supporting role, to create new magic that the human mind has yet to imagine, we will need to take creativity seriously well beyond the conventional pillars of more data, more compute, and more time on inference. That may be where the frontier labs currently are, but if you know about open-endedness, you know that open-ended creativity is not about taking a test and getting a good score. Lila understands the difference and is willing to invest to help me create the best open-endedness team in the world (as part of their overall AI effort) to make it happen. So I want to invite you, if you think this opportunity sounds as exciting as I do, to reach out if you’re interested in joining my team. I’m looking for a diverse range of expertise: pre-training, fine-tuning, RLHF, distillation, mechanistic interpretability, and yes - quality diversity techniques! We’re going to do things on my team that nobody else is doing. This will not be the usual roadmap. Compensation will be industry competitive and our team will be based in San Francisco with a hybrid work schedule. 1/n
0:35

Important announcement (with job opportunities!): I’m thrilled to share that I just joined Lila Sciences as SVP of Open-Endedness! Lila is a new name in the AI space, but one you will be hearing a lot from. Their unique mission to pursue Scientific Superintelligence could not align better with my interest in open-ended creativity. Lila is about the entire scientific enterprise, not just a single constrained domain like drug discovery. And of course, the history of scientific progress is an unmistakable precedent for open-ended discovery and Why Greatness Cannot Be Planned. For AI itself to join this epic quest beyond just a supporting role, to create new magic that the human mind has yet to imagine, we will need to take creativity seriously well beyond the conventional pillars of more data, more compute, and more time on inference. That may be where the frontier labs currently are, but if you know about open-endedness, you know that open-ended creativity is not about taking a test and getting a good score. Lila understands the difference and is willing to invest to help me create the best open-endedness team in the world (as part of their overall AI effort) to make it happen. So I want to invite you, if you think this opportunity sounds as exciting as I do, to reach out if you’re interested in joining my team. I’m looking for a diverse range of expertise: pre-training, fine-tuning, RLHF, distillation, mechanistic interpretability, and yes - quality diversity techniques! We’re going to do things on my team that nobody else is doing. This will not be the usual roadmap. Compensation will be industry competitive and our team will be based in San Francisco with a hybrid work schedule. 1/n

Kenneth Stanley

63,202 views • 1 year ago

Yann LeCun -- Meta's chief AI scientist and a Turing Award winner -- explains why scaling up LLMs will never reach human-level AI, and names the four things today's systems still can't do: "If you think that we're going to get to human-level AI by just training on more data and scaling up LLMs, you're making a mistake." "If you're an investor and you invest in a company that told you we're going to get to human-level AI and PhD level by just training on more data and with a few tricks... that was probably not a good idea." "There are ideas about how to go forward and have systems that are capable of doing what every intelligent animal and human are capable of doing, and that current AI systems are not capable of doing." "Understanding the physical world, having persistent memory, and being able to reason and plan. Those are the four characteristics that need to be there." "That requires systems that can acquire common sense, that can learn from natural sensors like video, as opposed to just text." The entire market is priced on a straight line from bigger models to AGI. One of the people who invented deep learning is telling you the line doesn't reach -- and that the real bottleneck isn't compute, it's world models. If he's right, the winner isn't whoever stacks the most GPUs. It's whoever teaches a machine to learn from video the way a child does.
0:58

Yann LeCun -- Meta's chief AI scientist and a Turing Award winner -- explains why scaling up LLMs will never reach human-level AI, and names the four things today's systems still can't do: "If you think that we're going to get to human-level AI by just training on more data and scaling up LLMs, you're making a mistake." "If you're an investor and you invest in a company that told you we're going to get to human-level AI and PhD level by just training on more data and with a few tricks... that was probably not a good idea." "There are ideas about how to go forward and have systems that are capable of doing what every intelligent animal and human are capable of doing, and that current AI systems are not capable of doing." "Understanding the physical world, having persistent memory, and being able to reason and plan. Those are the four characteristics that need to be there." "That requires systems that can acquire common sense, that can learn from natural sensors like video, as opposed to just text." The entire market is priced on a straight line from bigger models to AGI. One of the people who invented deep learning is telling you the line doesn't reach -- and that the real bottleneck isn't compute, it's world models. If he's right, the winner isn't whoever stacks the most GPUs. It's whoever teaches a machine to learn from video the way a child does.

Karl Mehta

34,818 views • 2 months ago

New course! Generative AI with Large Language Models, created with Amazon Web Services and hosted on Coursera. This course goes deep into the technical foundations of LLMs and how to use them. You can sign up here: You’ll work through the full life-cycle of a generative AI project, and learn specific techniques like RLHF; zero-shot, one-shot, and few-shot learning with LLMs; advanced prompting frameworks like ReAct; even fine-tuning LLMs, and gain hands-on practice with all of these techniques. Instructors Antje Barth Chris Fregly Shelbee Eigenbrode and Mike G Chambers all do incredible Generative AI work at AWS, and have supported many companies to build creative LLM applications. They bring tremendous practical LLM expertise to this course. I'm confident you’ll finish this course with a deeper understanding of how LLMs work, and how to use them. I hope you enjoy the course!
1:57

New course! Generative AI with Large Language Models, created with Amazon Web Services and hosted on Coursera. This course goes deep into the technical foundations of LLMs and how to use them. You can sign up here: You’ll work through the full life-cycle of a generative AI project, and learn specific techniques like RLHF; zero-shot, one-shot, and few-shot learning with LLMs; advanced prompting frameworks like ReAct; even fine-tuning LLMs, and gain hands-on practice with all of these techniques. Instructors Antje Barth Chris Fregly Shelbee Eigenbrode and Mike G Chambers all do incredible Generative AI work at AWS, and have supported many companies to build creative LLM applications. They bring tremendous practical LLM expertise to this course. I'm confident you’ll finish this course with a deeper understanding of how LLMs work, and how to use them. I hope you enjoy the course!

Andrew Ng

468,024 views • 3 years ago

RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!
0:37

RLHF by hand ✍️ ~ 15 steps walkthrough below Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so. RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move. But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned. How does it work? Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself. = 1. Given = A reward model, an LLM, and two (prompt, next) pairs. = 2. Preferences = A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal. = 3. Word embeddings = Let us look up each word of the loser pair. These vectors are the reward model's input. = 4. Linear layer = We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position. = 5. Mean pool = Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding. = 6. Output layer = We map that sentence down to a single number. Reward = 3. = 7. The winner, the same way = Let us repeat steps 3 to 6 on the winning pair. Reward = 5. = 8. Winner minus loser = We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it. = 9. Loss gradient = Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained. = 10. A prompt it has never seen = We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised. = 11. Transformer = Let us push it through attention and a feed forward layer, one vector per position. = 12. Output probabilities = We map each vector to a score over the vocabulary. = 13. Sample = Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2. = 14. Score it with the reward model = We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred. = 15. Loss gradient = Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights. The outputs: Loser reward = 3, winner reward = 5 Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1 LLM samples "him", reward = 3, LLM gradient = -1 Congrats! You just calculated RLHF by hand. And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted. 💾 Save this post!

Tom Yeh

22,529 views • 2 months ago

This AI can read emotions better than you can. It was created by Hume (Hume AI) an AI research lab developing models that can read your face and your voice with uncanny accuracy. Their hope is that models that can read your emotions will help create AI that optimizes for human well-being. I sat down with Alan Cowen (Alan Cowen), the co-founder and CEO of Hume to talk about how all of this works: the science of emotion, AI that optimizes for human well-being, and more. Before starting Hume, Alan helped set up Google’s research into affective computing and got a Ph.D. in computational psychology from Berkeley. He's one of the bright lights in AI, and this was an incredible conversation. We get into: - What an emotion actually is - Why traditional psychological theories of emotion are inadequate - How Hume is able to model human emotions - How Hume's API enables developers to build empathetic voice interfaces - Applications of the model in customer service, gaming, and therapy - Why Hume is designed to optimize for human well-being instead of engagement - The ethical concerns around creating an AI that can interpret human emotions - The future of psychology as a science This is a must-watch for anyone interested in the science of emotion and the future of human-AI interactions. Watch! --- Timestamps: I tell Hume’s empathetic AI model a secret: 00:00:00 Introduction: 00:01:13 What traditional psychology tells us about emotions: 00:10:17 Alan’s radical approach to studying human emotion: 00:13:46 Methods that Hume’s AI model uses to understand emotion: 00:16:46 How the model accounts for individual differences: 00:21:08 My pet theory on why it’s been hard to make progress in psychology: 00:27:19 The ways in which Alan thinks Hume can be used: 00:38:12 How Alan is thinking about the API v. consumer product question: 00:41:22
56:12

This AI can read emotions better than you can. It was created by Hume (Hume AI) an AI research lab developing models that can read your face and your voice with uncanny accuracy. Their hope is that models that can read your emotions will help create AI that optimizes for human well-being. I sat down with Alan Cowen (Alan Cowen), the co-founder and CEO of Hume to talk about how all of this works: the science of emotion, AI that optimizes for human well-being, and more. Before starting Hume, Alan helped set up Google’s research into affective computing and got a Ph.D. in computational psychology from Berkeley. He's one of the bright lights in AI, and this was an incredible conversation. We get into: - What an emotion actually is - Why traditional psychological theories of emotion are inadequate - How Hume is able to model human emotions - How Hume's API enables developers to build empathetic voice interfaces - Applications of the model in customer service, gaming, and therapy - Why Hume is designed to optimize for human well-being instead of engagement - The ethical concerns around creating an AI that can interpret human emotions - The future of psychology as a science This is a must-watch for anyone interested in the science of emotion and the future of human-AI interactions. Watch! --- Timestamps: I tell Hume’s empathetic AI model a secret: 00:00:00 Introduction: 00:01:13 What traditional psychology tells us about emotions: 00:10:17 Alan’s radical approach to studying human emotion: 00:13:46 Methods that Hume’s AI model uses to understand emotion: 00:16:46 How the model accounts for individual differences: 00:21:08 My pet theory on why it’s been hard to make progress in psychology: 00:27:19 The ways in which Alan thinks Hume can be used: 00:38:12 How Alan is thinking about the API v. consumer product question: 00:41:22

Dan Shipper 📧

45,941 views • 2 years ago

Why AI Can Now Make Discoveries - my conversation with Dan Roberts, Lead of the Foundations of Reinforcement Learning team at OpenAI 00:00 Intro: AI's wild week in mathematics 01:21 What OpenAI's Foundations of RL team does 03:08 Dan's journey: from black holes and quantum gravity to frontier AI 07:04 Are AI systems becoming useful for real science 08:21 The AI math moment: Erdős, OpenAI, DeepMind, and Anthropic 08:52 Why the OpenAI result was an act of exploration 10:25 OpenAI vs. DeepMind: informal reasoning vs. formal proof 12:13 RL 101: learning by doing, not just watching 15:10 Why reinforcement learning works 15:58 How RL breaks: sparse feedback and long-horizon tasks 17:03 RLHF: how human feedback shaped early language models 18:48 Move 37, self-play, and the search for novel strategies 22:16 Explore vs. exploit in scientific discovery 24:49 Why RL may now be "the cake," not the cherry on top 25:46 Why RL started working with large language models 27:29 Is RL "sucking supervision through a straw"? 28:47 Why language may be the grounding layer for intelligence 31:46 A contrarian take on the Bitter Lesson 32:41 What test-time compute actually is 34:50 How RL gives models the ability to think 35:40 Verifiable rewards, math, coding, and the messy real world 38:00 What physics can teach us about AI 42:08 Is there a thermodynamics of AI? 43:08 From Erdős problems to Einstein-level AI 45:16 Is AI already doing original science? 45:51 How far are we from AI automating AI research 47:41 Why Dan is excited about the future of science
49:06

Why AI Can Now Make Discoveries - my conversation with Dan Roberts, Lead of the Foundations of Reinforcement Learning team at OpenAI 00:00 Intro: AI's wild week in mathematics 01:21 What OpenAI's Foundations of RL team does 03:08 Dan's journey: from black holes and quantum gravity to frontier AI 07:04 Are AI systems becoming useful for real science 08:21 The AI math moment: Erdős, OpenAI, DeepMind, and Anthropic 08:52 Why the OpenAI result was an act of exploration 10:25 OpenAI vs. DeepMind: informal reasoning vs. formal proof 12:13 RL 101: learning by doing, not just watching 15:10 Why reinforcement learning works 15:58 How RL breaks: sparse feedback and long-horizon tasks 17:03 RLHF: how human feedback shaped early language models 18:48 Move 37, self-play, and the search for novel strategies 22:16 Explore vs. exploit in scientific discovery 24:49 Why RL may now be "the cake," not the cherry on top 25:46 Why RL started working with large language models 27:29 Is RL "sucking supervision through a straw"? 28:47 Why language may be the grounding layer for intelligence 31:46 A contrarian take on the Bitter Lesson 32:41 What test-time compute actually is 34:50 How RL gives models the ability to think 35:40 Verifiable rewards, math, coding, and the messy real world 38:00 What physics can teach us about AI 42:08 Is there a thermodynamics of AI? 43:08 From Erdős problems to Einstein-level AI 45:16 Is AI already doing original science? 45:51 How far are we from AI automating AI research 47:41 Why Dan is excited about the future of science

Matt Turck

69,801 views • 3 months ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.
0:48

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,353 views • 3 years ago

Web scraping is a critical skill, and yet nobody talks about it. How do you think companies are training their Large Language Models? Where do you think the data comes from? But web scraping goes beyond all of that. Imagine giving an AI agent access to any public online data in real time! I like to call this "web-scrapping on demand", and I'm pretty sure it's going to unlock unlimited power for AI applications. I recorded a quick video to show you how you can do this using Apify. I've talked about them before, and they are collaborating with me on this post. They have one of the best open-source web scraping and browser automation libraries out there: But it gets much better than this! You can use MCP to connect your AI Agents and applications to the Apify platform and use any specialized actor on demand to scrape and process online data. In the video, I used Cursor to scrape LinkedIn posts with the words "Machine Learning" in real time. Worked like a charm with no code needed! Here is a link to the platform: Think about this: You can now feed your AI applications with any public data on demand! We aren't ready for what's coming.
5:38

Web scraping is a critical skill, and yet nobody talks about it. How do you think companies are training their Large Language Models? Where do you think the data comes from? But web scraping goes beyond all of that. Imagine giving an AI agent access to any public online data in real time! I like to call this "web-scrapping on demand", and I'm pretty sure it's going to unlock unlimited power for AI applications. I recorded a quick video to show you how you can do this using Apify. I've talked about them before, and they are collaborating with me on this post. They have one of the best open-source web scraping and browser automation libraries out there: But it gets much better than this! You can use MCP to connect your AI Agents and applications to the Apify platform and use any specialized actor on demand to scrape and process online data. In the video, I used Cursor to scrape LinkedIn posts with the words "Machine Learning" in real time. Worked like a charm with no code needed! Here is a link to the platform: Think about this: You can now feed your AI applications with any public data on demand! We aren't ready for what's coming.

Santiago

101,473 views • 1 year ago

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.
1:05

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

237,774 views • 22 days ago

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.
2:52

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 views • 2 years ago