Загрузка видео...
Не удалось загрузить видео
Reasoning from scratch round 3: This time, I cover generating a verifier for... a) ...evaluation (base model versus any future model improvement) b) ...the reinforcement learning with verifiable rewards (RLVR) training later on 00:00 Introduction 01:21 Four approaches to LLM evaluation 07:20 Verifiers and reinforcement learning with verifiable rewards... show more
43,229 просмотров • 2 месяцев назад •via X (Twitter)
Комментарии: 37
And the YouTube version:
Glad you gave the verifier this much attention, Sebastian. For anyone following along, keep a few deliberately wrong answers in your tests too. It is reassuring to see the grader reject something you already know should fail.
Yes I recommend that too. In the demo I have a couple of both
Amazing!
four approaches to RLVR verification is a dense framing
4 approaches to LLM *evaluation*
Verifiers make reasoning improvements measurable.
verifier before rlvr. otherwise the reward just learns your eval bug
Verifiers are also quite useful for establishing performance baselines and comparing different models
the verifier needs its own eval set. if the same check grades progress and supplies the reward, one bug can make both dashboards agree while the model gets worse.
That is totally true. The RLVR chapters uses a non-overlapping dataset
The verifier starts as a referee and ends up inside the game. Keep adversarial holdouts outside the training loop, version every rule, and track disagreement with human graders—or RLVR may just get better at pleasing the scoreboard.
这个从零搭推理验证器的内容太实用了，刚好最近在做大模型推理优化，正需要这种从环境到实现的完整教程。
Building the math verifier before the RLVR loop is the order that keeps the training honest. Eval scaffolding that can grade future models beats another leaderboard screenshot.
The verifier-first framing holds outside training too. Once generation is cheap, the scarce part is a check you can run without a human reading everything. Most teams I see still have a reviewer, not a verifier.
The part that's underrated: verifier-as-evaluator and verifier-as-reward-signal fail in opposite directions. A weak eval verifier just gives you noisy numbers; a weak reward verifier actively teaches the policy to game it.
Deep dive Verifiers RLVR next level eval
A verifier that serves both evaluation and RLVR creates a clean bridge between measuring progress and training for it. The verifier quality becomes the whole game.
The evaluation verifier is the harder half. A reward verifier only has to be right on the distribution you train on, while a scorer has to keep holding as the policy drifts away from it.
the edge cases around answer extraction may matter more than the grader logic itself
Do you track answer-extraction failures separately from mathematical errors? With a verifier used for both RLVR and evaluation, a gain could partly reflect more parseable outputs. A small audit of unparseable answers would help distinguish the two.
The gap between those two at 1:10:47 often tells you more about your eval than your model does.
Steps 3.4 through 3.6 are where verifiers usually get lost: extract the box, normalize, then compare. Each of those steps buys coverage and spends the ability to say no. Normalize aggressively enough and two different answers compare equal — and the verifier still returns a number, it just is not measuring anything anymore. What keeps that honest is a negative control set: answers that must fail, each with a stated reason, plus an assertion that they failed for that reason. A cell that went red because the extractor threw is not evidence that equivalence checking works; it is a test that never exercised the check. The same question shows up one layer up, where the party being checked can also propose the checks. We put the verifier on the caller's side — orca exec --verifier "cargo test" "fix it" — so the agent cannot finish without passing something it did not choose and cannot renegotiate. Curious how you pick the leniency threshold in 3.5. That knob usually decides whether a verifier is real.
the boxed-answer grader is the whole eval. did MATH-500 scores hold when you swapped the prompt template, or is that the sensitivity bit?
Base models are super sensitive to prompt templates
yeah that's the sensitivity bit. a MATH-500 delta on a base model can be the template, not the verifier. freeze the template before you treat the boxed-answer score as the baseline
normalizing extracted answers before verifying is where most of these pipelines quietly break
评测口和 RLVR 口如果不是同一套 verifier，后面奖励会自己打架。从零做数学等价检查，比再堆一轮 SFT 更值钱。
everyone's chasing reasoning, nobody wants to build the verifier. the boring part is the whole game.
The base-model-versus-future-model use is the underrated half. Most teams only build a verifier once they're doing RL, so they have no way to answer "did last week's change help" — which is the question they get asked every week.
When reasoning model generates answer a verifier evaluates the answer and gives the evaluation score and generates the output Using Reinforcement learning with verifiable rewards (RLVR) it says that evaluate the answer and if it is correct give evaluation score as well as RL reward for it . This Reward helps model to generate more answers like that for that given input
why not start with a verifier for the *interview loop*? most models fail there because they can't handle the chaos of human feedback. how do you even define that?
This is the bit people skip. A verifier turns "looks good" into a pass/fail you can trust next week.
Wow this is so amazing!! Thanks for sharing
Interesting approach looking forward to how the verifier scales with future model updates. 🧠🚀
the verifier is doing double duty here: eval now, reward signal later. that bridge is where most RLVR demos get hand-wavy
I’d keep extraction failures visible beside the maths score. If a prompt change makes the answer easier to parse, the score can rise without the model getting better at the maths.
