Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

DPO Debate: Is RL needed for RLHF? All things as we cannot settle if DPO or RL is better. At least it is a good exercise. 1. Derivations in the DPO paper. Hint, the authors are good at math 2. cDPO, IPO, and related equations 3. Speculation on potential... show more

Nathan Lambert

89,218 subscribers

100,027 Aufrufe • vor 2 Jahren •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

11 Kommentare

Profilbild von Nathan Lambert

Nathan Lambertvor 2 Jahren

here we go again @abacaj @ericmitchellai @srush_nlp @_lewtun @hamishivi @rajammanabrolu @teortaxesTex @yacineMTB @erhartford @Teknium1 @ethayarajh

Profilbild von Allan Zhou

Allan Zhouvor 2 Jahren

If the derivation of Eq 4 is a bit esoteric, there's always the more direct (but tedious) approach: form the Lagrangian L(π)=E[r(y)]-βKL(π||π_ref)+λ(∑π(y)-1) and set dL/dπ(y)=0, solve.

Profilbild von Nathan Lambert

Nathan Lambertvor 2 Jahren

Can you post that derivation too?

Profilbild von Pengyu Cheng

Pengyu Chengvor 2 Jahren

Nice summary！BTW, please also take a glance at our APO (Adversarial Preference Optimization) at which is an adversarial alignment method and compatible to either DPO/IPO/PPO😃

Profilbild von RanW

RanWvor 2 Jahren

“Language tasks are trees” I finally found it! Second guessing: what about goal-oriented dialogues, having the ability to backspace, etc.

Profilbild von Nathan Lambert

Nathan Lambertvor 2 Jahren

Goal oriented as you describe it prolly works, but I'm not a theory person! Good point!

Profilbild von Mohammad Azar

Mohammad Azarvor 2 Jahren

@ylecun @natolambert have you read the IPO paper carefully? We had chats with main authors of DPO and they are aware of limitations of DPO, i.e., Bradly Theory assumption and overfitting. DPO is a pioneering work in this field and in IPO we try to improve it by alleviating limitations.

Profilbild von Nathan Lambert

Nathan Lambertvor 2 Jahren

@ylecun I read it a good bit, but definitely not all. Two things stuck out that made me turn away from complete read: * It came off as pretty aggressive towards DPO, which is a different style of paper to read * It still needs a lot of experiments I'm happy to keep learning more

Profilbild von Snowflect

Snowflectvor 2 Jahren

@ylecun Don’t forget the #GazaGenocide and #IsraelCriminalWar and ongoing atrocities

Profilbild von Mohammad Azar

Mohammad Azarvor 2 Jahren

I am not sure expressing quite negative comments about a work (based on softballs like "there is no experiments, which is not true") either help the advancement of the field or help people gain better understanding of the topic.

Profilbild von Nathan Lambert

Nathan Lambertvor 2 Jahren

you're right, followed up via email.

Ähnliche Videos

JUSTICE FOR UMMUKHAIRI The Kaduna State Commissioner of Police went on the Berekete Family show to boldly deny that the DPO released Ummukhairi to the bloodthirsty mob that brutally murdered her. But lies cannot erase the truth Look at this video evidence. It clearly captures the exact moment the DPO escorted her out and handed her directly over to her killers. He delivered her straight to her death! We are urgently calling on the Kaduna State Government and the Federal Government of Nigeria to act immediately. This DPO must not go free! Bring him to justice now #JusticeForUmmukhairi

JUSTICE FOR UMMUKHAIRI The Kaduna State Commissioner of Police went on the Berekete Family show to boldly deny that the DPO released Ummukhairi to the bloodthirsty mob that brutally murdered her. But lies cannot erase the truth Look at this video evidence. It clearly captures the exact moment the DPO escorted her out and handed her directly over to her killers. He delivered her straight to her death! We are urgently calling on the Kaduna State Government and the Federal Government of Nigeria to act immediately. This DPO must not go free! Bring him to justice now #JusticeForUmmukhairi

Fateema Zahra✨🌟

612,695 Aufrufe • vor 1 Monat

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

h100envy

490,734 Aufrufe • vor 22 Tagen

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 Aufrufe • vor 1 Jahr

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

New Course: Post-training of LLMs Learn to post-train and customize an LLM in this short course, taught by Banghua Zhu, Assistant Professor at the University of Washington University of Washington, and co-founder of @NexusflowX. Training an LLM to follow instructions or answer questions has two key stages: pre-training and post-training. In pre-training, it learns to predict the next word or token from large amounts of unlabeled text. In post-training, it learns useful behaviors such as following instructions, tool use, and reasoning. Post-training transforms a general-purpose token predictor—trained on trillions of unlabeled text tokens—into an assistant that follows instructions and performs specific tasks. Because it is much cheaper than pre-training, it is practical for many more teams to incorporate post-training methods into their workflows than pre-training. In this course, you’ll learn three common post-training methods—Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning (RL)—and how to use each one effectively. With SFT, you train the model on pairs of input and ideal output responses. With DPO, you provide both a preferred (chosen) and a less preferred (rejected) response and train the model to favor the preferred output. With RL, the model generates an output, receives a reward score based on human or automated feedback, and updates the model to improve performance. You’ll learn the basic concepts, common use cases, and principles for curating high-quality data for effective training. Through hands-on labs, you’ll download a pre-trained model from Hugging Face and post-train it using SFT, DPO, and RL to see how each technique shapes model behavior. In detail, you’ll: - Understand what post-training is, when to use it, and how it differs from pre-training. - Build an SFT pipeline to turn a base model into an instruct model. - Explore how DPO reshapes behavior by minimizing contrastive loss—penalizing poor responses and reinforcing preferred ones. - Implement a DPO pipeline to change the identity of a chat assistant. - Learn online RL methods such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), and how to design reward functions. - Train a model with GRPO to improve its math capabilities using a verifiable reward. Post-training is one of the most rapidly developing areas of LLM training. Whether you’re building a high-accuracy context-specific assistant, fine-tuning a model's tone, or improving task-specific accuracy, this course will give you experience with the most important techniques shaping how LLMs are post-trained today. Please sign up here:

Andrew Ng

125,146 Aufrufe • vor 1 Jahr

Ex-JPMorgan engineer who wrote the LLM Course explained everything about fine-tuning and merging in 18 minutes - better than $2500 fine-tuning bootcamps. pick the base -> LoRA or QLoRA -> then DPO or KTO for alignment -> merge two fine-tunes into one stronger model -> ship a model that beats the base on your task. That loop is why Labonne's merges sit at the top of the Hugging Face leaderboard. LoRA + QLoRA + DPO + KTO + mergekit - that's the stack. Watch and save it, then merge your first two fine-tunes this week.

Ex-JPMorgan engineer who wrote the LLM Course explained everything about fine-tuning and merging in 18 minutes - better than $2500 fine-tuning bootcamps. pick the base -> LoRA or QLoRA -> then DPO or KTO for alignment -> merge two fine-tunes into one stronger model -> ship a model that beats the base on your task. That loop is why Labonne's merges sit at the top of the Hugging Face leaderboard. LoRA + QLoRA + DPO + KTO + mergekit - that's the stack. Watch and save it, then merge your first two fine-tunes this week.

h100envy

13,856 Aufrufe • vor 16 Tagen

I watched in horror as the DPO dragged her away. I warned him “If u take her outside that police station, the mob will kill her.” He ignored the warning, handed her over to the crowd, and and they burned her alive. This is the painful reality , the very institution entrusted with protecting our lives & property is now being accused of delivering innocent citizens to their death. What is even more heartbreaking is that many have already moved on Safeeya nd Ahamad marital issues, while some high ranking officers in Kaduna are reportedly shielding this DPO instead of demanding justice. We must not forget. We must not be silent. We must demand accountability. #JusticeForHer #EndMobViolence #AccountabilityNow

I watched in horror as the DPO dragged her away. I warned him “If u take her outside that police station, the mob will kill her.” He ignored the warning, handed her over to the crowd, and and they burned her alive. This is the painful reality , the very institution entrusted with protecting our lives & property is now being accused of delivering innocent citizens to their death. What is even more heartbreaking is that many have already moved on Safeeya nd Ahamad marital issues, while some high ranking officers in Kaduna are reportedly shielding this DPO instead of demanding justice. We must not forget. We must not be silent. We must demand accountability. #JusticeForHer #EndMobViolence #AccountabilityNow

Extraordinary Ustaz 👳🏾‍♂️📿

23,536 Aufrufe • vor 1 Monat

The Kano state Government must make sure the killers of the DPO are brought to justice, it’s not only when it happens in the south that everyone shout and cry for justice. The Uromi Massacre live rent free in my head and I can remember how quick the Kano state government put fire on Edo state government to bring the killers to justice. The boys involved in the Uromi massacre were all arrested within 48hours. Same energy must be put into arresting the boys involved in killing of the DPO. We must not hear that they are unknown Kano Fulani Hunters.

The Kano state Government must make sure the killers of the DPO are brought to justice, it’s not only when it happens in the south that everyone shout and cry for justice. The Uromi Massacre live rent free in my head and I can remember how quick the Kano state government put fire on Edo state government to bring the killers to justice. The boys involved in the Uromi massacre were all arrested within 48hours. Same energy must be put into arresting the boys involved in killing of the DPO. We must not hear that they are unknown Kano Fulani Hunters.

Mr Ben 🇳🇬 🇨🇦 🇬🇧

31,471 Aufrufe • vor 1 Jahr

OpenAI engineer explained every form of fine-tuning in 1 hour 46 minutes - better than $3000 fine-tuning bootcamps. pick your task -> try prompting first -> drop to SFT for style -> run DPO for preference -> run RFT for verifiable rewards -> ship a model 10x cheaper than GPT-5. That loop is why teams that fine-tune beat teams that only prompt on cost and speed. Prompting + SFT + DPO + RFT + OpenAI fine-tuning API - that's the stack. Watch and save it, then pick the right fine-tune for your task this week.

OpenAI engineer explained every form of fine-tuning in 1 hour 46 minutes - better than $3000 fine-tuning bootcamps. pick your task -> try prompting first -> drop to SFT for style -> run DPO for preference -> run RFT for verifiable rewards -> ship a model 10x cheaper than GPT-5. That loop is why teams that fine-tune beat teams that only prompt on cost and speed. Prompting + SFT + DPO + RFT + OpenAI fine-tuning API - that's the stack. Watch and save it, then pick the right fine-tune for your task this week.

h100envy

40,941 Aufrufe • vor 15 Tagen

UPDATE: Emergency services arrived on time at the scene. All 3 people trapped have been rescued to the hospital. All in stable condition. The collapsed building has been barricaded by authorities. . Special thanks to the Ogun State Police Command, DSP OLUSEYI BABASEYI, the DPO of Mowe, Ogun State Fire Service, and other emergency services for the quick response.

UPDATE: Emergency services arrived on time at the scene. All 3 people trapped have been rescued to the hospital. All in stable condition. The collapsed building has been barricaded by authorities. . Special thanks to the Ogun State Police Command, DSP OLUSEYI BABASEYI, the DPO of Mowe, Ogun State Fire Service, and other emergency services for the quick response.

Man of Letters.

14,174 Aufrufe • vor 3 Monaten

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

The DeepSeek-R1 paper is a gem! Highly encourage everyone to read it. It's clear that LLM reasoning capabilities can be learned in different ways. RL, if applied correctly and at scale, can lead to some really powerful and interesting scaling and emergent properties. There is more to RL than meets the eye! Here is my breakdown of the paper along with a few tests: The multi-state training might not make sense initially but they provide clues on optimizations that we can continue to tap into. Data quality is still very important for enhancing the usability of the LLM. Unlike other reasoning LLMs, DeepSeek-R1's training recipe and weights are open so we can build on top of it. This opens up exciting research opportunities. About the attached clip: the previous preview model wasn't able to solve this task. DeepSeek-R1 can solve this and many other tasks that o1 can solve. It's a very good model for coding and math.

elvis

140,692 Aufrufe • vor 1 Jahr

Reinforcement Learning from Human Feedback (RLHF) is gaining traction. This field aims to make AI more responsible by including human values and preferences. In this video, Nathan Lambert, a research scientist and RLHF team lead at Hugging Face explores its inner workings, applications and industry impact. RLHF has gained the spotlight in recent years. The growth of language models like Anthropic’s Claude and OpenAI's ChatGPT have increased interest in human-feedback integration. "There are some rumors that Open AI had two teams; one was doing RLHF and the other instruction fine-tuning. And the RLHF team kept getting more and more performance." Understanding RLHF The RLHF process has three main steps: Pre-training: Much like with GPT models, the journey starts with pre-training on a large corpus of data. This can range from text data, web scrapes, to specialized datasets. Reward Modeling: This is the RLHF counterpart of supervised fine-tuning in large language models. This stage involves creating a reward model that resonates with human values and preferences. RL Optimization: This stage parallels reward modeling and reinforcement learning in traditional AI models. The AI system fine-tunes itself based on the reward model, employing reinforcement learning algorithms for that extra layer of optimization. The Data Challenge Data collection and curation in RLHF closely resemble the challenges you'd encounter in large language model training. Datasets from organizations like OpenAI can serve as a useful foundation. However, the need for high-quality, task-specific data cannot be overstated. Implementing RLHF: A Practical Guide If you’re someone who loves getting hands-on with AI libraries like Hugging Face, implementing RLHF is right way to do. It’s essential to understand its limitations. Think about model stability, over-optimization, and exploration strategies, much like you would when prompt engineering. Ongoing Research and Next Steps While he suggests that some basics figured out, there are layers of complexity that still need to be unraveled: 1. New Benchmarks: How do we measure the effectiveness of RLHF? 2. Preference Modeling: How can the model be made to understand human preferences better? 3. Interpreting RLHF: Much like explainability in traditional models, how do we make RLHF more interpretable? 4. System-Wide Evaluation: Going beyond individual performance, how does RLHF affect an entire system? The Transformative Power of RLHF Whether you're an AI developer, a business analyst, or a marketer, RLHF promises to revolutionize your domain. Imagine customer service chatbots that understand human emotions better, or content generators that align more closely with human values. RLHF is an emerging field that focuses on enhancing machine learning models through human feedback. While it tackles important issues like bias and ethics, its broader goal is to improve system performance across various applications. Whether you're deeply invested in the ethics of AI or simply curious about advancements in machine learning, RLHF offers valuable insights. If you're interested in the next wave of AI development, this area is definitely one to watch.

Reinforcement Learning from Human Feedback (RLHF) is gaining traction. This field aims to make AI more responsible by including human values and preferences. In this video, Nathan Lambert, a research scientist and RLHF team lead at Hugging Face explores its inner workings, applications and industry impact. RLHF has gained the spotlight in recent years. The growth of language models like Anthropic’s Claude and OpenAI's ChatGPT have increased interest in human-feedback integration. "There are some rumors that Open AI had two teams; one was doing RLHF and the other instruction fine-tuning. And the RLHF team kept getting more and more performance." Understanding RLHF The RLHF process has three main steps: Pre-training: Much like with GPT models, the journey starts with pre-training on a large corpus of data. This can range from text data, web scrapes, to specialized datasets. Reward Modeling: This is the RLHF counterpart of supervised fine-tuning in large language models. This stage involves creating a reward model that resonates with human values and preferences. RL Optimization: This stage parallels reward modeling and reinforcement learning in traditional AI models. The AI system fine-tunes itself based on the reward model, employing reinforcement learning algorithms for that extra layer of optimization. The Data Challenge Data collection and curation in RLHF closely resemble the challenges you'd encounter in large language model training. Datasets from organizations like OpenAI can serve as a useful foundation. However, the need for high-quality, task-specific data cannot be overstated. Implementing RLHF: A Practical Guide If you’re someone who loves getting hands-on with AI libraries like Hugging Face, implementing RLHF is right way to do. It’s essential to understand its limitations. Think about model stability, over-optimization, and exploration strategies, much like you would when prompt engineering. Ongoing Research and Next Steps While he suggests that some basics figured out, there are layers of complexity that still need to be unraveled: 1. New Benchmarks: How do we measure the effectiveness of RLHF? 2. Preference Modeling: How can the model be made to understand human preferences better? 3. Interpreting RLHF: Much like explainability in traditional models, how do we make RLHF more interpretable? 4. System-Wide Evaluation: Going beyond individual performance, how does RLHF affect an entire system? The Transformative Power of RLHF Whether you're an AI developer, a business analyst, or a marketer, RLHF promises to revolutionize your domain. Imagine customer service chatbots that understand human emotions better, or content generators that align more closely with human values. RLHF is an emerging field that focuses on enhancing machine learning models through human feedback. While it tackles important issues like bias and ethics, its broader goal is to improve system performance across various applications. Whether you're deeply invested in the ethics of AI or simply curious about advancements in machine learning, RLHF offers valuable insights. If you're interested in the next wave of AI development, this area is definitely one to watch.

Muratcan Koylan

27,005 Aufrufe • vor 2 Jahren

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Satpal Singh Rathore

46,010 Aufrufe • vor 11 Monaten

3 Arrested – Stolen Vehicle DCSO Deputies were patrolling Highlands Ranch when they reviewed an alert from an Automatic License Plate Reader for a stolen vehicle. Deputies began searching the area for the car and located it parked in a strip mall’s lot. Deputies formulated a plan and blocked the vehicle in the parking spot. Deputies took the driver and rear passenger into custody without incident, then located the 3rd passenger walking the strip mall's parking lot and took her into custody without incident. During the search of the vehicle and subjects in custody, narcotics were found, including paraphernalia. The driver (resident of Longmont) was arrested for: •Motor Vehicle Theft (F4) •Unlawful Possession of a controlled substance (DM1) •Driving on a revoked license (MT2) •Possession of Drug Paraphernalia (DPO) The front passenger (resident of Aurora) was arrested for: •Unlawful Possession of a controlled substance more than 4 grams (DF4) •Criminal Impersonation (F6) •2 Felony Warrants •1 Misdemeanor Warrant •Possession of Drug Paraphernalia (DPO The rear passenger (resident of Denver) was arrested for: •Unlawful Possession of a controlled substance (DM1) •Possession of Drug Paraphernalia (DPO) Great teamwork by DCSO deputies in safely taking them all into custody and preventing a pursuit!

3 Arrested – Stolen Vehicle DCSO Deputies were patrolling Highlands Ranch when they reviewed an alert from an Automatic License Plate Reader for a stolen vehicle. Deputies began searching the area for the car and located it parked in a strip mall’s lot. Deputies formulated a plan and blocked the vehicle in the parking spot. Deputies took the driver and rear passenger into custody without incident, then located the 3rd passenger walking the strip mall's parking lot and took her into custody without incident. During the search of the vehicle and subjects in custody, narcotics were found, including paraphernalia. The driver (resident of Longmont) was arrested for: •Motor Vehicle Theft (F4) •Unlawful Possession of a controlled substance (DM1) •Driving on a revoked license (MT2) •Possession of Drug Paraphernalia (DPO) The front passenger (resident of Aurora) was arrested for: •Unlawful Possession of a controlled substance more than 4 grams (DF4) •Criminal Impersonation (F6) •2 Felony Warrants •1 Misdemeanor Warrant •Possession of Drug Paraphernalia (DPO The rear passenger (resident of Denver) was arrested for: •Unlawful Possession of a controlled substance (DM1) •Possession of Drug Paraphernalia (DPO) Great teamwork by DCSO deputies in safely taking them all into custody and preventing a pursuit!

DC Sheriff

14,073 Aufrufe • vor 8 Monaten

Building upon SimpleVLA-RL, we have implemented real-world RL on long-horizon dexterous tasks and witnessed a non-trivial (~relatively 300%) performance improvement over the SFT model, along with surprising capabilities on auto-recovery. Blog coming soon. The entire process uses very little data and training compute—basically costing no more than a single robotic arm—hinting that real-world generality for machines is actually within sight.

Building upon SimpleVLA-RL, we have implemented real-world RL on long-horizon dexterous tasks and witnessed a non-trivial (~relatively 300%) performance improvement over the SFT model, along with surprising capabilities on auto-recovery. Blog coming soon. The entire process uses very little data and training compute—basically costing no more than a single robotic arm—hinting that real-world generality for machines is actually within sight.

Ning Ding

93,278 Aufrufe • vor 6 Monaten

RL is a powerful mechanism for training company-specific models on their unique work and data. This is what we do at Applied Compute. A key challenge is how to make RL efficient, because we need runs to be fast (delivered in days), cheap (scalable unit economics), and predictable (not just fast, but reliably fast). Here are some takeaways: • Synchronous RL is wasteful with time and compute. • Asynchronous RL is more efficient but introduces staleness, which causes learning instabilities. • Modeling and simulations can help analytically solve for what configuration leads to optimal efficiency. This allows us to rapidly prototype training configurations, without burning expensive compute cycles on trial runs. Two of our co-founders, Rhythm Garg and Linden Li, discussed some of this research at AI Engineer recently, with a focus on the following subproblem: what is the highest throughput way to do RL given a maximum staleness and compute budget?

RL is a powerful mechanism for training company-specific models on their unique work and data. This is what we do at Applied Compute. A key challenge is how to make RL efficient, because we need runs to be fast (delivered in days), cheap (scalable unit economics), and predictable (not just fast, but reliably fast). Here are some takeaways: • Synchronous RL is wasteful with time and compute. • Asynchronous RL is more efficient but introduces staleness, which causes learning instabilities. • Modeling and simulations can help analytically solve for what configuration leads to optimal efficiency. This allows us to rapidly prototype training configurations, without burning expensive compute cycles on trial runs. Two of our co-founders, Rhythm Garg and Linden Li, discussed some of this research at AI Engineer recently, with a focus on the following subproblem: what is the highest throughput way to do RL given a maximum staleness and compute budget?

Applied Compute

45,555 Aufrufe • vor 7 Monaten

New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.

New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.

Nathan Lambert

92,467 Aufrufe • vor 1 Monat

Imagine coming home to invest and then this happens to you 💔💔💔 “I just returned, acquired some land at home. One of the things I think I could contribute to the community is fully fledged mechanised farming, on which I invested… we bought the equipments from China. The DPO came to my office and said we did not take permission from him to discharge the equipments in our warehouse, and as a result, they took me to the police station.” - Chief Pius Ileogben

Imagine coming home to invest and then this happens to you 💔💔💔 “I just returned, acquired some land at home. One of the things I think I could contribute to the community is fully fledged mechanised farming, on which I invested… we bought the equipments from China. The DPO came to my office and said we did not take permission from him to discharge the equipments in our warehouse, and as a result, they took me to the police station.” - Chief Pius Ileogben

The Benin Blogger

63,050 Aufrufe • vor 25 Tagen

MC Okrika is trending because the frustrated comedian confronted rogue police officers in Etung, Ikom, Cross Rivers who harrased, extorted ₦150k from a farmer carrying moi moi and pure water. He’s also accusing the DPO of complicity in illegal border smuggling. A young farmer and his brother were going to farm with moi moi and pure water for lunch, when the officers stopped them at the checkpoint asking what they were carrying. He told them it’s food, water and moi moi. The officers insisted they must search the bag; since they weren’t wearing a tagged uniform, the farmer insisted to record them as they searched his bags — all hell broke loose. The police officer became violent and started beating him for questioning their job and “illegally” recording them. They then threatened to kill him in the bush. As the assault escalated, his brother intervened and pulled the officers away. The officers then recorded this part, and sent it to the DPO of Etung Divisional HQ, who immediately ordered the arrest of the two brothers for “fighting” an officer on duty, and later collected ₦150k from their family for their bail. Interestingly, the same officer who refused to be recorded while “searching” the moi moi bag is the one Mc Okrika caught with a big wrap of Indian Hemp today at the checkpoint in Etung. Okrika also alleged that the officers are notorious for planting contraband on unsuspecting commuters and extorting them on the directives of the DPO, who was also at the checkpoint in mufti. The State Commissioner of Police, CP Rashid Afegbua, has reacted to the incident and vowed to investigate and discipline all officers involved. Follow Trending Explained for daily explanations!

MC Okrika is trending because the frustrated comedian confronted rogue police officers in Etung, Ikom, Cross Rivers who harrased, extorted ₦150k from a farmer carrying moi moi and pure water. He’s also accusing the DPO of complicity in illegal border smuggling. A young farmer and his brother were going to farm with moi moi and pure water for lunch, when the officers stopped them at the checkpoint asking what they were carrying. He told them it’s food, water and moi moi. The officers insisted they must search the bag; since they weren’t wearing a tagged uniform, the farmer insisted to record them as they searched his bags — all hell broke loose. The police officer became violent and started beating him for questioning their job and “illegally” recording them. They then threatened to kill him in the bush. As the assault escalated, his brother intervened and pulled the officers away. The officers then recorded this part, and sent it to the DPO of Etung Divisional HQ, who immediately ordered the arrest of the two brothers for “fighting” an officer on duty, and later collected ₦150k from their family for their bail. Interestingly, the same officer who refused to be recorded while “searching” the moi moi bag is the one Mc Okrika caught with a big wrap of Indian Hemp today at the checkpoint in Etung. Okrika also alleged that the officers are notorious for planting contraband on unsuspecting commuters and extorting them on the directives of the DPO, who was also at the checkpoint in mufti. The State Commissioner of Police, CP Rashid Afegbua, has reacted to the incident and vowed to investigate and discipline all officers involved. Follow Trending Explained for daily explanations!

Trending Explained

12,275 Aufrufe • vor 2 Monaten

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"

Dwarkesh Patel

502,249 Aufrufe • vor 8 Monaten

Another friday afternoon talk in attempt of open science. This is a re-record of a talk I gave at two Chantham House Rules workshops this fall: The History and Risks of Reinforcement Learning and Human Feedback (yes, the and is clever and intentional). This tries to broaden the discussion around RLHF to include more stakeholders and diverse ideas about how to integrate or measure values (can you at all?!?). I'm trying to re-record every talk that I give that isn't to a public audience 🫡😅, good practice at least. Slides: Youtube: Paper:

Another friday afternoon talk in attempt of open science. This is a re-record of a talk I gave at two Chantham House Rules workshops this fall: The History and Risks of Reinforcement Learning and Human Feedback (yes, the and is clever and intentional). This tries to broaden the discussion around RLHF to include more stakeholders and diverse ideas about how to integrate or measure values (can you at all?!?). I'm trying to re-record every talk that I give that isn't to a public audience 🫡😅, good practice at least. Slides: Youtube: Paper:

Nathan Lambert

54,036 Aufrufe • vor 2 Jahren