Video yükleniyor...
Video Yüklenemedi
Reasoning From Scratch: Reinforcement Learning with Verifiable Rewards (RLVR) round 2. Covering clipped policy ratios, KL loss term, format rewards, and other GRPO tips & tricks. 00:00 Introduction and recap 01:52 Interpreting basic GRPO training metrics 06:34 Planned improvements to GRPO 08:58 Running longer training jobs with Python scripts... show more
60,891 görüntüleme • 1 ay önce •via X (Twitter)
35 Yorum

And here's the YouTube version:

ok the clipped policy ratio section is the one i'm queueing up first

The clipped policy ratio part finally clicked for me here. I mostly work on the app side, and this makes the training behavior feel a lot less like magic. Thanks for putting it together.

Glad this was useful!

connecting the equations to the logs is the draw here. implementing a loss is one thing, knowing how to read the run takes more explanation.

the order of the chapter is the lesson. you plot the metrics and diagnose the instability first, then add each fix after you can see why it's needed

saving this, grpo still confuses me so this helps a lot

kinda hot take but i'd cut the kl term first for rlvr. dapo dropped it entirely and the runs seemed fine without it

Yes, that’s what I did :)

The reasoning from scratch series is pure gold. I am learning a lot from you @rasbt 🙏

Format rewards deserve more attention in GRPO write-ups. If they are weighted too heavily, the policy learns the template before the task, and the reward curve still looks healthy. Logging answer accuracy separately from format compliance makes that failure visible early.

clipped policy ratios make sense for stability, but how do verifiable rewards scale when the verification itself becomes the bottleneck? does RLVR address latency at inference?

@bargava We talked about RLVR once on our podcast. This seems like a great technical exposition on the subject.

Just wanna say thanks for amazing learning material, your work on dissecting how LLMs and modern AI work is amazing!

Thanks :)

@grok what's one thing RL with verifiable rewards can learn that human ratings can't teach?

Glad entropy tracking is in here. Most tutorials skip how to tell when training is about to collapse.

saw the intro and thought it set the tone well. always tricky balancing clarity with complexity. reminds me of when the 3rd period gets too lost in jargon. just keep it straightforward or it’s like running in circles, right?

rewarding format only for correct answers is the difference between reasoning and cosplay

Round two on the same topic says the first one worked. Is the KL term the bit people get wrong most often?

Not necessarily but it adds more complexity and is not necessary for some domains (like math)

The GRPO tips and tricks are the real payload — clipped policy ratios and the KL loss term are where most reimplementations quietly die, since that's where theory meets training-dynamics reality. Round 2 this fast says the RLVR playbook is maturing faster than the discourse.

Thanks for this workshop, so much useful stuff packed in here

@rasbt Fixing GRPO with KL plus clipped ratios after reward hacking is the real loop. When MATH-500 dips, do you roll back the checkpoint or keep training with tighter clips?

Curious how much it helped to give the format reward only on correct answers. Was the model gaming the think tags before you switched?

the 'diagnosing unstable training' chapter is the one i'd skip to first. that's where most people quietly give up

Remember when reward hacking was just a buzzword? Seeing KL penalties now feels like full circle.

Once the KL term is in, does entropy still collapse on longer runs or does that mostly go away

Clipped policy ratios were the ones that that helped most with that in this case

I’m still learning GRPO. If format reward only applies to correct answers, how do you keep an early run moving when correct answers are rare? Is that mainly a starting-checkpoint issue, or do you change the reward setup?

The model (training) should work fine without format reward, it’s just an optional stylistic thing. If the model never gets the answers right then you’d need to start with a simpler dataset.

does the clipped ratio stay stable for you when the verifier rewards are sparse, or does the kl term end up doing most of the work?

Advantage stats and entropy in the same walkthrough. That helps connect the equations to what the run is doing.

答案错了,就算它把“思考过程”写得再像样,也别给格式分。不然模型可能忙着学怎么显得很会想,题还是做不对。

The underappreciated half of RLVR: 'verifiable' just moves the bottleneck to the ground truth. Your reward signal is only as clean as the data behind the verifier — garbage verifications, garbage gradients.


