Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks. 00:00 Introduction and recap 01:52 Interpreting basic GRPO training metrics 06:34 Planned improvements to GRPO 08:58 Running longer training jobs with Python scripts...

60,891 görüntüleme • 1 ay önce •via X (Twitter)

35 Yorum

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

And here's the YouTube version:

minime · fireply.ai profil fotoğrafı
minime · fireply.ai1 ay önce

ok the clipped policy ratio section is the one i'm queueing up first

Sachin profil fotoğrafı
Sachin1 ay önce

The clipped policy ratio part finally clicked for me here. I mostly work on the app side, and this makes the training behavior feel a lot less like magic. Thanks for putting it together.

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

Glad this was useful!

Jeff Bruchado profil fotoğrafı
Jeff Bruchado1 ay önce

connecting the equations to the logs is the draw here. implementing a loss is one thing, knowing how to read the run takes more explanation.

Fireply.ai profil fotoğrafı
Fireply.ai1 ay önce

the order of the chapter is the lesson. you plot the metrics and diagnose the instability first, then add each fix after you can see why it's needed

BreezeOg profil fotoğrafı
BreezeOg1 ay önce

saving this, grpo still confuses me so this helps a lot

Saksham Jain profil fotoğrafı
Saksham Jain1 ay önce

kinda hot take but i'd cut the kl term first for rlvr. dapo dropped it entirely and the runs seemed fine without it

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

Yes, that’s what I did :)

Dean Harron profil fotoğrafı
Dean Harron1 ay önce

The reasoning from scratch series is pure gold. I am learning a lot from you @rasbt 🙏

0xss.eth profil fotoğrafı
0xss.eth1 ay önce

Format rewards deserve more attention in GRPO write-ups. If they are weighted too heavily, the policy learns the template before the task, and the reward curve still looks healthy. Logging answer accuracy separately from format compliance makes that failure visible early.

Jatin Garg profil fotoğrafı
Jatin Garg1 ay önce

clipped policy ratios make sense for stability, but how do verifiable rewards scale when the verification itself becomes the bottleneck? does RLVR address latency at inference?

aiexplorations profil fotoğrafı
aiexplorations1 ay önce

@bargava We talked about RLVR once on our podcast. This seems like a great technical exposition on the subject.

Armin Catovic profil fotoğrafı
Armin Catovic1 ay önce

Just wanna say thanks for amazing learning material, your work on dissecting how LLMs and modern AI work is amazing!

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

Thanks :)

zeus profil fotoğrafı
zeus1 ay önce

@grok what's one thing RL with verifiable rewards can learn that human ratings can't teach?

Subham Das profil fotoğrafı
Subham Das1 ay önce

Glad entropy tracking is in here. Most tutorials skip how to tell when training is about to collapse.

Nolan profil fotoğrafı
Nolan1 ay önce

saw the intro and thought it set the tone well. always tricky balancing clarity with complexity. reminds me of when the 3rd period gets too lost in jargon. just keep it straightforward or it’s like running in circles, right?

Logan profil fotoğrafı
Logan1 ay önce

rewarding format only for correct answers is the difference between reasoning and cosplay

Marius Laurusevicius profil fotoğrafı
Marius Laurusevicius1 ay önce

Round two on the same topic says the first one worked. Is the KL term the bit people get wrong most often?

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

Not necessarily but it adds more complexity and is not necessary for some domains (like math)

Next Billion Labs profil fotoğrafı
Next Billion Labs1 ay önce

The GRPO tips and tricks are the real payload — clipped policy ratios and the KL loss term are where most reimplementations quietly die, since that's where theory meets training-dynamics reality. Round 2 this fast says the RLVR playbook is maturing faster than the discourse.

Blum profil fotoğrafı
Blum1 ay önce

Thanks for this workshop, so much useful stuff packed in here

Rohan Bhanot profil fotoğrafı
Rohan Bhanot1 ay önce

@rasbt Fixing GRPO with KL plus clipped ratios after reward hacking is the real loop. When MATH-500 dips, do you roll back the checkpoint or keep training with tighter clips?

AI Quanting profil fotoğrafı
AI Quanting1 ay önce

Curious how much it helped to give the format reward only on correct answers. Was the model gaming the think tags before you switched?

Rafael Ibayev profil fotoğrafı
Rafael Ibayev1 ay önce

the 'diagnosing unstable training' chapter is the one i'd skip to first. that's where most people quietly give up

IronRed | SandHive profil fotoğrafı
IronRed | SandHive1 ay önce

Remember when reward hacking was just a buzzword? Seeing KL penalties now feels like full circle.

Manu | 🥥 profil fotoğrafı
Manu | 🥥1 ay önce

Once the KL term is in, does entropy still collapse on longer runs or does that mostly go away

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

Clipped policy ratios were the ones that that helped most with that in this case

Biplove Yadav | AI Engineer profil fotoğrafı
Biplove Yadav | AI Engineer1 ay önce

I’m still learning GRPO. If format reward only applies to correct answers, how do you keep an early run moving when correct answers are rare? Is that mainly a starting-checkpoint issue, or do you change the reward setup?

Sebastian Raschka profil fotoğrafı
Sebastian Raschka1 ay önce

The model (training) should work fine without format reward, it’s just an optional stylistic thing. If the model never gets the answers right then you’d need to start with a simpler dataset.

Dominik profil fotoğrafı
Dominik1 ay önce

does the clipped ratio stay stable for you when the verifier rewards are sparse, or does the kl term end up doing most of the work?

Somil Singh | Inside AI profil fotoğrafı
Somil Singh | Inside AI1 ay önce

Advantage stats and entropy in the same walkthrough. That helps connect the equations to what the run is doing.

韩策 profil fotoğrafı
韩策1 ay önce

答案错了,就算它把“思考过程”写得再像样,也别给格式分。不然模型可能忙着学怎么显得很会想,题还是做不对。

Daltaris profil fotoğrafı
Daltaris1 ay önce

The underappreciated half of RLVR: 'verifiable' just moves the bottleneck to the ground truth. Your reward signal is only as clean as the data behind the verifier — garbage verifications, garbage gradients.

Benzer Videolar

Better Than ElevenLabs? Ultimate TTS Voice Training & Cloning in 600+ Languages Watch on YouTube : Chapters: 0:00 My trained AI voice 0:38 600+ supported languages 1:19 Windows and cloud options 1:46 Your first voice clone 2:10 Generate and listen 3:18 Download ZIP and guides 3:46 Windows Requirements 4:28 Choose an install path 4:40 Extract the package 4:54 Run the installer 5:23 PyTorch, CUDA and wheels 5:54 Start the application 6:19 Models, voices, GPU presets 7:04 First load vs warm runs 7:46 Players and saved outputs 8:34 IndexTTS English example 9:02 AuK English example 9:52 Reference library and files 10:48 Select a reference range 11:38 Microphone recording 11:53 Reference text and transcription 13:19 Clone and Auto voice 14:38 Voice design tags 15:26 Language selection 15:40 Preview and Smart sentences 16:32 Section-length study 17:01 Sentence and section pauses 18:08 Pause tags 19:04 Check and set pronunciation 19:38 Save and reuse dictionary 21:10 Expressive tags 21:44 Keep which take 21:58 Fewest word errors 22:12 Rank takes and set checks 22:42 Ranking and fallback result 23:49 Whole-text candidates 24:36 60-line selection study 25:05 139-line Auto voice study 25:35 First vs selected take 26:42 Matching-language references 27:26 Arabic voice example 28:24 Chinese voice example 29:04 French voice example 29:46 German voice example 30:40 Italian voice example 31:22 Japanese voice example 32:03 Korean voice example 32:58 Polish voice example 33:38 Spanish voice example 33:49 Cross-language cloning 35:48 Speaking-rate comparison 36:49 Target duration modes 37:39 Best quality: 32 steps 38:25 Fast: 16 steps 38:54 Seeds and advanced sampling 39:44 WAV, MP3 and filenames 40:01 Reference and subtitle export 40:16 Inspect the exported files 41:10 Recent outputs as references 41:52 Audio tuning comparison 43:09 Tuning and edge silence 43:30 SRT import and cue timing 44:22 Create a narrated MP4 45:08 Batch inputs and filenames 45:43 Shared and per-file voices 46:07 Batch execution and progress 46:47 Inspect batch results 48:00 Save a personal preset 48:18 Load settings and reference 49:26 Recent settings and theme 49:45 Select GPU and memory tier 50:02 BF16 and ConvRot INT8 50:16 Measured memory use 50:30 Measured synthesis time 50:44 Compare both variants 52:25 Section batch size 52:39 Load models and find files 53:03 VRAM benchmark and tier cap 53:56 Train a new OmniVoice 54:15 Source recordings and text 54:44 Reserve evaluation sessions 55:00 Dataset folder and scan 55:37 Transcript and alignment 55:57 Segment duration 56:13 Padding and caption cleanup 56:28 Quality filters 56:47 Prepare and inspect clips 58:02 Speaker and transcript audit 58:50 Review flagged clips 59:21 Correct words and boundaries 1:00:20 Build the OmniVoice cache 1:00:47 Dataset and training name 1:01:06 Full fine-tuning 1:01:33 Full, LoRA and DoRA 1:01:50 Held-out voice comparison 1:02:05 Epochs and Training plan 1:02:22 Token budget and accumulation 1:02:39 Learning rate and memory 1:03:14 Checkpoints and train state 1:03:30 Continue vs weights-only 1:03:47 Evaluation and final test 1:04:18 Fluency analysis and filters 1:05:16 Start training 1:05:49 Speed, loss and validation 1:06:18 Hear a training sample 1:06:45 Stop and resume controls 1:07:05 800-update completion 1:07:39 Checkpoint analysis 1:07:55 Automatic speech comparison 1:08:15 Reference audition 1:08:37 Speaking-rate calibration 1:08:53 Save an INT8 voice copy 1:09:46 Load the trained preset 1:10:14 Generate held-out speech 1:11:12 Base vs trained voice 1:12:15 Configure Checkpoint Grid 1:13:24 Hear the six grid cells 1:14:34 Trained Auto voice 1:15:22 Trained INT8 voice 1:16:13 Updates, backup and help 1:17:01 Your next voice

Furkan Gözükara

60,722 görüntüleme • 1 ay önce

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,697 görüntüleme • 1 yıl önce