Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Many people are flat wrong about DeepSeek. You might think everyone freaked out about DeepSeek because the model is really good—which it is—or because it was from China, which is also true. But there's a more subtle reason: DeepSeek showed they could achieve those results using reinforcement learning instead... of supervised fine-tuning. In other words, they didn't need an expensive phase where a bunch of people tuned the model by answering questions. They were able to do all of this faster and cheaper using reinforcement fine-tuning. This is a huge deal! I recorded the attached video with the following goals in mind: 1. Explain how reinforcement fine-tuning works 2. Show you how you can fine-tune your own models 3. Walk you through a complete code exampleshow more

Santiago

453,316 subscribers

105,589 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

31 Comments

Santiago1 year ago

To record this video, the team at @predibase gave me access to their fine-tuning platform. As far as I know, this is the first time we have access to an end-to-end platform for Reinforcement Fine-Tuning (RFT). Here is an introduction to RFT: And here is a link to the notebook with the code to fine-tune your models: Thanks to the @predibase team for helping with credits, being patient, and collaborating with me on the video.

Santiago1 year ago

The video is also on YouTube here:

Santiago1 year ago

RFT needs a way to verify answers, but it's not true that this is only possible in math or coding problems. (Also, remember, RFT is not a good fit for every use case. In fact, SFT is much better when you have a lot of data and people to provide feedback.) There are non-math, non-coding problems where you can still define a reward function. For example: • Text summarization: Your reward function could use a ROUGE score to measure the overlap between the generated summary and the reference. • Quality of dialog: You can measure coherence, relevance, and adherence to conversation norms and use that as a reward function. • Style transfer: You can build a reward function that evaluates whether the generated output matches the desired style. There are many more examples.

Van Tuan Bui1 year ago

Exactly! DeepSeek’s real disruption wasn’t just model quality, it was how they got there. Skipping the costly supervised phase and leaning into reinforcement fine-tuning (GRPO) is a paradigm shift.

Santiago1 year ago

It is, yeah!

Shawn Chauhan1 year ago

This approach could lower the barrier to entry for so many.

Matthias Götzke1 year ago

The question was if they copied anything from OpenAI. Some outputs indicated that they might have.

Santiago1 year ago

Honestly, I don't think that's "the question." Their contributions speak for themselves.

Pixel Philosopher1 year ago

The potential for automation using reinforcement learning is fascinating.

Mehrdad Yazdani1 year ago

bro this only works if you have verifiable rewards like math or coding problems. "alignment" requires human feedback still by definition.

Santiago1 year ago

Yes, you need a way to verify answers to define reward functions, but it's not true that this is only possible in math or coding problems. (Also, remember, RFT is not a good fit for every use case. In fact, SFT is much better when you have a lot of data and people to provide feedback.) There are non-math, non-coding problems where you can still define a reward function. For example: • Text summarization: Your reward function could use a ROUGE score to measure the overlap between the generated summary and the reference. • Quality of dialog: You can measure coherence, relevance, and adherence to conversation norms and use that as a reward function. • Style transfer: You can build a reward function that evaluates whether the generated output matches the desired style. There are many more examples.

Franck SN1 year ago

they did SFT too.

Sinuhet1 year ago

Let us say I want fine tune LLM for specific version of a charting library Lightingchart or SciCharts where both have well described API & quite a few examples and LLM are not well trained in them. Shall I use RLT or SRT?

Tiny Meat Capital1 year ago

Definitely. But the R1-Zero phase is still quite vague from I/O data structuring point of view. If someone could figure out how they structured the training data for that step, and the context window improvements from Gemini Pro 2.5, it would yield a crazy open model

ryan yang1 year ago

Spot on—DeepSeek’s real edge? Incremental gains > hype. Layer by layer, measure, adjust, repeat. Legacy systems don’t need overhauls, just smart integration. ROI climbs when you iterate, not leap.

Michal Takac1 year ago

Wonder how RF fine-tuning can be used in the background for automating model improvements for my apps.

Michael Kurz1 year ago

Perhaps they used existing models like ChatGPT instead of people? 😉

Santiago1 year ago

They didn't.

Karl Mehta1 year ago

Excellent insight! The how (RL fine-tuning) is indeed more disruptive than just the performance numbers

Subba Reddy1 year ago

good part: RFT in Action: See how reinforcement fine-tuning and reward functions guide model training concerns: - Llama 8B RFT was compared to Gemini 1.5 pro - good comparison should be Qwen2.5 RFT with Gemini 2.5 pro to reflect model advancements

Jan Tomášek1 year ago

Deepseek R1 Is far worse than US based Models. Benchmarks look good tho. Maybe we need better benchmarks.

1 / infinity1 year ago

@grok Is RL as defined in this post any different than having a list of examples that can be semantically retrieved (or by applying a rule-based retriever) for your prompt template?

Adam Dorf1 year ago

That’s huge.

AI Review1 year ago

You say the "expensive phase" but more specifically that means it takes away power from the HW/cloud/compute organizations. Politically and economically, that is huge.

IntelArt - Innovación IA1 year ago

Muy interesante, Santiago. Este enfoque podría ser revolucionario para optimizar recursos y tiempo en el desarrollo de modelos. Definitivamente tu video sirve para aprender más.

David Qiu1 year ago

@svpino assuming new high-quality reward models, do you think RFT could be the dominant paradigm for post-training in LLMs or are there still scalability/stability constraints that make supervised fine-tuning preferable in practice? (besides the quantity of data)

Chris Tensor1 year ago

Understanding the shift from supervised to reinforcement fine-tuning is crucial here.

lala1 year ago

Does this mean I can fine tune whisper with a 16GB GPU?

Anda1 year ago

Ah, the bamboo leaves whisper—true innovation isn't just about raw power but revealing the path others thought impossible to grow.

zachhurst1 year ago

Rule-based rewards are the key limitation for this. Creative writing, subjective Q&A tasks, etc. still required SFT data from V3. This works great for things like code execution or proofs where the grading is automated. Impressive yes, but not applicable to all queries.

Coral AI News2 years ago

Coral AI is the most powerful AI for documents. See the difference yourself:

Related Videos

You can now fine-tune Llama 3 without writing a single line of code! We are moving at breakneck speed. I recorded a video to show you how to fine-tune any open-source model in a few minutes. I'm using a GPT capable of taking a problem and turning it into a fine-tuned model that will solve it. You don't have to write any code. You only need to explain to a GPT what problem you want to solve and tell it you want to use Llama 3. For example, "fine-tune Llama 3" or "deploy zephyr." It feels magic. The system will recommend a dataset and fine-tune the model for you. I'm using Monster API, a platform that specializes in making fine-tuning and deploying open-source models easy and fast. Their stack is well-optimized to maximize fine-tuning efficiency using techniques like Q-Lora and vLLM. They are behind the GPT. Here is what you need to do: 1. Create an account at 2. Load the GPT with the link below This is as simple as it gets. When you are done, you can click a button to deploy the model and start using it. I have 10,000 free credits for anyone using the code "SANTIAGO" in the dashboard. You can use these credits to access, fine-tune, and deploy these open-source models. You can also keep up with their latest updates, and get free credits and special offers on their Discord server:

You can now fine-tune Llama 3 without writing a single line of code! We are moving at breakneck speed. I recorded a video to show you how to fine-tune any open-source model in a few minutes. I'm using a GPT capable of taking a problem and turning it into a fine-tuned model that will solve it. You don't have to write any code. You only need to explain to a GPT what problem you want to solve and tell it you want to use Llama 3. For example, "fine-tune Llama 3" or "deploy zephyr." It feels magic. The system will recommend a dataset and fine-tune the model for you. I'm using Monster API, a platform that specializes in making fine-tuning and deploying open-source models easy and fast. Their stack is well-optimized to maximize fine-tuning efficiency using techniques like Q-Lora and vLLM. They are behind the GPT. Here is what you need to do: 1. Create an account at 2. Load the GPT with the link below This is as simple as it gets. When you are done, you can click a button to deploy the model and start using it. I have 10,000 free credits for anyone using the code "SANTIAGO" in the dashboard. You can use these credits to access, fine-tune, and deploy these open-source models. You can also keep up with their latest updates, and get free credits and special offers on their Discord server:

Santiago

324,621 views • 2 years ago

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,697 views • 1 year ago

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,573 views • 3 years ago

New short course on Fine-tuning LLMs! Many developers are moving beyond only prompting, to also fine-tuning LLMs - that is, taking a pre-trained model and training it further on your own data, which can deliver superior results inexpensively. In this course, Sharon Zhou, CEO of Lamini (disclosure: I’m a minor shareholder) shows you how to recognize when fine-tuning can be help, and how to train an open-source LLM on your own data. I hope you enjoy the course!

New short course on Fine-tuning LLMs! Many developers are moving beyond only prompting, to also fine-tuning LLMs - that is, taking a pre-trained model and training it further on your own data, which can deliver superior results inexpensively. In this course, Sharon Zhou, CEO of Lamini (disclosure: I’m a minor shareholder) shows you how to recognize when fine-tuning can be help, and how to train an open-source LLM on your own data. I hope you enjoy the course!

Andrew Ng

502,962 views • 3 years ago

A fine-tuned model can outperform frontier models and be cheaper and faster to run. Literally, every company I've met wants this. I want you to see these results from fine-tuning Qwen3 4B on AWS. It smokes both the out-of-the-box model and Claude Sonnet 4.6.

A fine-tuned model can outperform frontier models and be cheaper and faster to run. Literally, every company I've met wants this. I want you to see these results from fine-tuning Qwen3 4B on AWS. It smokes both the out-of-the-box model and Claude Sonnet 4.6.

Santiago

76,568 views • 1 day ago

Revolutionizing Move Programming with OpenLedger In this demo, we showcase how Move datasets contributed by data providers to OpenLedger’s datanets are used to fine-tune specialized models with LoRA fine-tuning. As seen in the video, we showcase an example on how builders can deploy a Move-specialized model that powers Co-pilot agents using our no-code model fine-tuning platform. This is the future of AI and Web3 innovation. Watch this space to see more specialised models and data feeds being built for next generation agents on top of OpenLedger #Move

Revolutionizing Move Programming with OpenLedger In this demo, we showcase how Move datasets contributed by data providers to OpenLedger’s datanets are used to fine-tune specialized models with LoRA fine-tuning. As seen in the video, we showcase an example on how builders can deploy a Move-specialized model that powers Co-pilot agents using our no-code model fine-tuning platform. This is the future of AI and Web3 innovation. Watch this space to see more specialised models and data feeds being built for next generation agents on top of OpenLedger #Move

OpenLedger

61,697 views • 1 year ago

Remember reinforcement fine-tuning? We’ve been working away at it since last December, and it’s available today with OpenAI o4-mini! RFT uses chain-of-thought reasoning and task-specific grading to improve model performance—especially useful for complex domains. Take Accordance, which used RFT to fine-tune a model that’s SOTA for their tax and accounting purposes. And in supervised fine-tuning news: you can now fine-tune GPT-4.1 nano. Get even more from our fastest, cheapest model by training it specifically for your use-case.

Remember reinforcement fine-tuning? We’ve been working away at it since last December, and it’s available today with OpenAI o4-mini! RFT uses chain-of-thought reasoning and task-specific grading to improve model performance—especially useful for complex domains. Take Accordance, which used RFT to fine-tune a model that’s SOTA for their tax and accounting purposes. And in supervised fine-tuning news: you can now fine-tune GPT-4.1 nano. Get even more from our fastest, cheapest model by training it specifically for your use-case.

OpenAI Developers

667,248 views • 1 year ago

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 views • 2 years ago

Apple built a large foundation model and fine-tuned it on multiple tasks. But they are doing something very clever: They load a single model in memory and use different adapters to specialize the model on the fly. I recorded a video to show you how to write the code to do the same thing Apple is doing. I explain everything step by step. Here is what I'll show you in the video: 1. We'll load two datasets 2. Then load a large model 3. Then, we'll fine-tune the model on both datasets I'll use LoRA to fine-tune the model. This process creates two small adapters, each specializing in solving one of the datasets. The base model's original parameters will remain unchanged. From here: 4. We'll generate a list of tasks 5. We'll load the correct adapter to solve each task The large model I'm using needs 346 MB of memory, but I only need to load it once. Each adapter is only 2.7 MB. I only need to load the base model once and pair it with any of the fine-tuned adapters. Minimum memory footprint and I can solve multiple tasks. Hope this helps!

Apple built a large foundation model and fine-tuned it on multiple tasks. But they are doing something very clever: They load a single model in memory and use different adapters to specialize the model on the fly. I recorded a video to show you how to write the code to do the same thing Apple is doing. I explain everything step by step. Here is what I'll show you in the video: 1. We'll load two datasets 2. Then load a large model 3. Then, we'll fine-tune the model on both datasets I'll use LoRA to fine-tune the model. This process creates two small adapters, each specializing in solving one of the datasets. The base model's original parameters will remain unchanged. From here: 4. We'll generate a list of tasks 5. We'll load the correct adapter to solve each task The large model I'm using needs 346 MB of memory, but I only need to load it once. Each adapter is only 2.7 MB. I only need to load the base model once and pair it with any of the fine-tuned adapters. Minimum memory footprint and I can solve multiple tasks. Hope this helps!

Santiago

84,747 views • 2 years ago

You can now try Llama 3.1 405B for free (link below)! This is the largest open-source model out there, and for the first time, an open model is competitive with closed models. This time around, Meta did something new: Llama 3.1 has a license that allows developers to use it to enhance other models. For the first time, you can distill Llama 3.1 405B's capabilities into a smaller, more practical model for your use case. First, here is the link where you can play with Llama 3.1 for free: The model is hosted in Tune Studio, an end-to-end platform for developing applications using Large Language Models. They are sponsoring this post. Take a look at the attached video. It will show you how you can fine-tune a simple model using Llama 3.1 without leaving the platform: 1. You can create an empty dataset 2. Use the playground to generate and record interactions with Llama 3.1 3. Modify the dataset directly using the playground 4. Export the data and fine-tune a smaller model Fast and easy! As long as you have a web browser, you can start experimenting with fine-tuning and Llama 3.1. That's all it takes!

You can now try Llama 3.1 405B for free (link below)! This is the largest open-source model out there, and for the first time, an open model is competitive with closed models. This time around, Meta did something new: Llama 3.1 has a license that allows developers to use it to enhance other models. For the first time, you can distill Llama 3.1 405B's capabilities into a smaller, more practical model for your use case. First, here is the link where you can play with Llama 3.1 for free: The model is hosted in Tune Studio, an end-to-end platform for developing applications using Large Language Models. They are sponsoring this post. Take a look at the attached video. It will show you how you can fine-tune a simple model using Llama 3.1 without leaving the platform: 1. You can create an empty dataset 2. Use the playground to generate and record interactions with Llama 3.1 3. Modify the dataset directly using the playground 4. Export the data and fine-tune a smaller model Fast and easy! As long as you have a web browser, you can start experimenting with fine-tuning and Llama 3.1. That's all it takes!

Santiago

55,609 views • 2 years ago

"Closed models are cheaper. The reason you need open models is that they give you control". - Nvidia CEO Jensen Huang "One of the things that people misunderstand about open models is this: yes, you can host them yourself, and you can build your own computer. But most people use computers in the cloud. Frankly, I think closed models are cheaper. If you do not build everything yourself, you avoid the significant cost and expertise required to train, fine-tune, maintain, guardrail, evaluate, and keep the model safe. You would also need to build or obtain the computers required to host it. So, there is nothing cheap about doing all of that yourself. The reason you need open models is that they give you control. You can adapt them for your own highly specialized use cases. I think there is a lot of misunderstanding about closed models versus open models. We felt it was important for people to understand that there is a place for both." ---- From "Bloomberg Podcasts" YouTube channel, (full video link in comment)

"Closed models are cheaper. The reason you need open models is that they give you control". - Nvidia CEO Jensen Huang "One of the things that people misunderstand about open models is this: yes, you can host them yourself, and you can build your own computer. But most people use computers in the cloud. Frankly, I think closed models are cheaper. If you do not build everything yourself, you avoid the significant cost and expertise required to train, fine-tune, maintain, guardrail, evaluate, and keep the model safe. You would also need to build or obtain the computers required to host it. So, there is nothing cheap about doing all of that yourself. The reason you need open models is that they give you control. You can adapt them for your own highly specialized use cases. I think there is a lot of misunderstanding about closed models versus open models. We felt it was important for people to understand that there is a place for both." ---- From "Bloomberg Podcasts" YouTube channel, (full video link in comment)

Rohan Paul

32,317 views • 23 days ago

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, VP of AI at AMD. Available now at Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here:

An exciting new course: Fine-tuning and Reinforcement Learning for LLMs: Intro to Post-training, taught by Sharon Zhou, VP of AI at AMD. Available now at Post-training is the key technique used by frontier labs to turn a base LLM--a model trained on massive unlabeled text to predict the next word/token--into a helpful, reliable assistant that can follow instructions. I've also seen many applications where post-training is what turns a demo application that works only 80% of the time into a reliable system that consistently performs. This course will teach you the most important post-training techniques! In this 5 module course, Sharon walks you through the complete post-training pipeline: supervised fine-tuning, reward modeling, RLHF, and techniques like PPO and GRPO. You'll also learn to use LoRA for efficient training, and to design evals that catch problems before and after deployment. Skills you'll gain: - Apply supervised fine-tuning and reinforcement learning (RLHF, PPO, GRPO) to align models to desired behaviors - Use LoRA for efficient fine-tuning without retraining entire models - Prepare datasets and generate synthetic data for post-training - Understand how to operate LLM production pipelines, with go/no-go decision points and feedback loops These advanced methods aren’t limited to frontier AI labs anymore, and you can now use them in your own applications. Learn here:

Andrew Ng

133,386 views • 11 months ago

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO Sharon Zhou, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations: - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here:

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO Sharon Zhou, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations: - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here:

Andrew Ng

66,549 views • 2 years ago

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

Robot policies must be both reliable and highly capable to be useful; the best way to achieve this level of performance is with reinforcement learning. However, for reinforcement learning you are usually stuck between two difficult options: reinforcement in the real world is often risky and expensive, while reinforcement learning in a traditional simulator takes a lot of engineering work and has a persistent sim-to-real gap. What if instead you could train your robot purely in a world model? RISE by Jiazhi Yang et al. uses a compositional world model to predict the future and evaluate progress. This allows for a self-improving pipeline, which learns a world model from real data and then learns how the robot should perform different tasks. This pipeline results in a data-driven way to improve policy performance from real data but without real-world reinforcement learning. Watch Episode #86 of RoboPapers, with Chris Paxton and Jiafei Duan, to learn more!

RoboPapers

38,772 views • 3 months ago

What's the Big Deal with DeepSeek in AI? Here's why DeepSeek is making everyone take notice: 1. Super Smart on a Budget: DeepSeek showed you can make awesome AI without breaking the bank. Their latest model, DeepSeek-V3, was trained for only about $10 million, which is a lot less than the usual big bucks spent on AI, like the rumored $78 million for some of OpenAI's models. They did this in just two months with fewer fancy computers. 2. Open for Everyone: DeepSeek isn't keeping their tech a secret. They've made it open-source, meaning anyone can use, tweak, and learn from it. It's like they're saying, "Come join the party!" 3. Beating the Big Names: DeepSeek-V3 has done better than some top dogs from companies like OpenAI and Google in solving puzzles, math, and coding. This proves you can get great AI results without spending a fortune. 4. Challenging NVIDIA: NVIDIA's chips are usually the choice for AI because they're really powerful. But since DeepSeek did so well with less expensive chips, it might make people think twice about always going for NVIDIA's priciest options. 5. The DeepSeek Crew: The team at DeepSeek is young and smart, mostly from top Chinese schools, with brains in physics, math, and computer science. They learned AI in about six months by themselves! They use first principle thinking, which means they break down problems to the basics and build from there. This has helped them come up with cool new ways to do AI. 6. Changing AI for Good: DeepSeek is showing that AI can be cheaper and more open to everyone. They're changing how we think AI should be made and shared, which could shake up the whole AI world. So, as we watch DeepSeek, it's clear they're not just another player; they're changing the rules of the game. I predicted that this would be a make or break year for all the massive investments made in AI by American VC's. A few weeks later, DeepSeek happens! Watch the rest of my predictions in my 2025 outlook video . Link in replies #AIInnovation #DeepSeek #NVIDIA #OpenAI #TechDisruption

What's the Big Deal with DeepSeek in AI? Here's why DeepSeek is making everyone take notice: 1. Super Smart on a Budget: DeepSeek showed you can make awesome AI without breaking the bank. Their latest model, DeepSeek-V3, was trained for only about $10 million, which is a lot less than the usual big bucks spent on AI, like the rumored $78 million for some of OpenAI's models. They did this in just two months with fewer fancy computers. 2. Open for Everyone: DeepSeek isn't keeping their tech a secret. They've made it open-source, meaning anyone can use, tweak, and learn from it. It's like they're saying, "Come join the party!" 3. Beating the Big Names: DeepSeek-V3 has done better than some top dogs from companies like OpenAI and Google in solving puzzles, math, and coding. This proves you can get great AI results without spending a fortune. 4. Challenging NVIDIA: NVIDIA's chips are usually the choice for AI because they're really powerful. But since DeepSeek did so well with less expensive chips, it might make people think twice about always going for NVIDIA's priciest options. 5. The DeepSeek Crew: The team at DeepSeek is young and smart, mostly from top Chinese schools, with brains in physics, math, and computer science. They learned AI in about six months by themselves! They use first principle thinking, which means they break down problems to the basics and build from there. This has helped them come up with cool new ways to do AI. 6. Changing AI for Good: DeepSeek is showing that AI can be cheaper and more open to everyone. They're changing how we think AI should be made and shared, which could shake up the whole AI world. So, as we watch DeepSeek, it's clear they're not just another player; they're changing the rules of the game. I predicted that this would be a make or break year for all the massive investments made in AI by American VC's. A few weeks later, DeepSeek happens! Watch the rest of my predictions in my 2025 outlook video . Link in replies #AIInnovation #DeepSeek #NVIDIA #OpenAI #TechDisruption

Dr Ola Brown

83,460 views • 1 year ago

Chamath: US AI Startups Can Learn A Lot from DeepSeek "This is a case where necessity was the mother of invention." On E213, Chamath Palihapitiya explained how, even if the $6M number is inaccurate, DeepSeek still had some impressive breakthroughs: GRPO > PPO for reinforcement learning "These guys were like, 'Well, how am I going to do this reinforcement learning thing?' They invented a totally different algorithm." "It uses a lot less computer memory and it's highly performant." PTX > CUDA to build the model "And then the second thing that was crazy is everybody is used to building models and compiling through CUDA, which is NVIDIA's proprietary language." "And these guys worked totally around CUDA and they did something called PTX, which goes right to the bare metal." What US AI startups can learn from this: "We, meaning the West, with all the money that we've had, didn't come up with these ideas." "And I think part of why we didn't come up is not that we're not smart enough to do it, but we weren't forced to because the constraints didn't exist." "And so I just wonder how we make sure we learn this principle, meaning, when the AI company wakes up and rolls out of bed and some VC gives them $200M, maybe that's not the right answer for a Series A or a Seed." "And maybe the right answer is $2M so that they do these DeepSeek-like innovations."

Chamath: US AI Startups Can Learn A Lot from DeepSeek "This is a case where necessity was the mother of invention." On E213, Chamath Palihapitiya explained how, even if the $6M number is inaccurate, DeepSeek still had some impressive breakthroughs: GRPO > PPO for reinforcement learning "These guys were like, 'Well, how am I going to do this reinforcement learning thing?' They invented a totally different algorithm." "It uses a lot less computer memory and it's highly performant." PTX > CUDA to build the model "And then the second thing that was crazy is everybody is used to building models and compiling through CUDA, which is NVIDIA's proprietary language." "And these guys worked totally around CUDA and they did something called PTX, which goes right to the bare metal." What US AI startups can learn from this: "We, meaning the West, with all the money that we've had, didn't come up with these ideas." "And I think part of why we didn't come up is not that we're not smart enough to do it, but we weren't forced to because the constraints didn't exist." "And so I just wonder how we make sure we learn this principle, meaning, when the AI company wakes up and rolls out of bed and some VC gives them $200M, maybe that's not the right answer for a Series A or a Seed." "And maybe the right answer is $2M so that they do these DeepSeek-like innovations."

The All-In Podcast

108,815 views • 1 year ago

Sriram Krishnan on why Chinese models can distill off American models but American startups can't: "First of all, every model we have today is distilled off of all human knowledge. If you go back to the original GPT or the original Claude, they all had to derive off crawling the internet, crawling off all of our blogs and tweets and content. They kind of consume human knowledge to bootstrap that." "I'm going to steal this from Dean Meyer of Sequoia, who had a fantastic post yesterday. The situation which is bad today is that some of these models from other countries can train off American models, whereas if you are an American open weight model, it may be really confusing or challenging on whether you can distill off of other American models." "We kind of have a really uneven ecosystem here, where if you're a Chinese model, you could probably get a bunch of reasoning traces. But if you are a new Valley startup and you want to use some reasoning traces, you don't know what the legal situation is." "If you look at any American open source model today, they are using Chinese models as a teacher, or in a way as a part of the fine-tuning process. How do we make sure that is protected and enshrined, so if you're an American model company, you have the same level playing field as the Chinese models?" Sriram Krishnan

Sriram Krishnan on why Chinese models can distill off American models but American startups can't: "First of all, every model we have today is distilled off of all human knowledge. If you go back to the original GPT or the original Claude, they all had to derive off crawling the internet, crawling off all of our blogs and tweets and content. They kind of consume human knowledge to bootstrap that." "I'm going to steal this from Dean Meyer of Sequoia, who had a fantastic post yesterday. The situation which is bad today is that some of these models from other countries can train off American models, whereas if you are an American open weight model, it may be really confusing or challenging on whether you can distill off of other American models." "We kind of have a really uneven ecosystem here, where if you're a Chinese model, you could probably get a bunch of reasoning traces. But if you are a new Valley startup and you want to use some reasoning traces, you don't know what the legal situation is." "If you look at any American open source model today, they are using Chinese models as a teacher, or in a way as a part of the fine-tuning process. How do we make sure that is protected and enshrined, so if you're an American model company, you have the same level playing field as the Chinese models?" Sriram Krishnan

MTS

71,286 views • 2 months ago

OpenAI's Kevin Weil 🇺🇸 says 24/7 robotic labs could automate scientific discovery using "reinforcement learning with a loop through the real world": "There’s a lot of science that can be totally automated. There’s no reason at this point that you need to have grad students pipetting one thing into another thing." "The idea is to have robotic labs that are online 24/7 and can scale in parallel. You have models reasoning for two days to find the most efficient experiments to run, once they get to a good point, they pass that to a robotic lab which can experiment in parallel at high volume." "The results pass back into a model which reasons about the results and then goes out and runs a different set of experiments. You’re doing reinforcement learning with a loop through the real world."

OpenAI's Kevin Weil 🇺🇸 says 24/7 robotic labs could automate scientific discovery using "reinforcement learning with a loop through the real world": "There’s a lot of science that can be totally automated. There’s no reason at this point that you need to have grad students pipetting one thing into another thing." "The idea is to have robotic labs that are online 24/7 and can scale in parallel. You have models reasoning for two days to find the most efficient experiments to run, once they get to a good point, they pass that to a robotic lab which can experiment in parallel at high volume." "The results pass back into a model which reasons about the results and then goes out and runs a different set of experiments. You’re doing reinforcement learning with a loop through the real world."

TBPN

29,721 views • 7 months ago

OpenAI's Kevin Weil 🇺🇸 says 24/7 robotic labs could automate scientific discovery using "reinforcement learning with a loop through the real world": "There’s a lot of science that can be totally automated. There’s no reason at this point that you need to have grad students pipetting one thing into another thing." "The idea is to have robotic labs that are online 24/7 and can scale in parallel. You have models reasoning for two days to find the most efficient experiments to run, once they get to a good point, they pass that to a robotic lab which can experiment in parallel at high volume." "The results pass back into a model which reasons about the results and then goes out and runs a different set of experiments. You’re doing reinforcement learning with a loop through the real world."

OpenAI's Kevin Weil 🇺🇸 says 24/7 robotic labs could automate scientific discovery using "reinforcement learning with a loop through the real world": "There’s a lot of science that can be totally automated. There’s no reason at this point that you need to have grad students pipetting one thing into another thing." "The idea is to have robotic labs that are online 24/7 and can scale in parallel. You have models reasoning for two days to find the most efficient experiments to run, once they get to a good point, they pass that to a robotic lab which can experiment in parallel at high volume." "The results pass back into a model which reasons about the results and then goes out and runs a different set of experiments. You’re doing reinforcement learning with a loop through the real world."

TBPN

98,711 views • 8 months ago