正在加载视频...
视频加载失败
New (shorter) lecture! Over-optimization, foundations of reward hacking, sycophancy, verbosity, etc. In recording this, I realized that rubrics are going to be prone to overopt in a way like reward models, where RLVR is its own thing. This is mostly fundamentals, history, and reflections! 00:00 Intro & Why We... show more
33,387 次观看 • 2 个月前 •via X (Twitter)
8 条评论

youtube:

the bad reputation part is deserved, half the field still thinks RLHF means one reward model and calls it a day

Timely

Seeing this on our closing-ops evals too. Once graders reward 'looks complete,' agents learn to satisfy the rubric, not the closing. How do you stop a rubric from becoming the thing gamed rather than a proxy for real work?

Spot on about rubrics, they break way faster than standard reward models

i see this behaviour all the time... I accidentally make a false claim to GPT but it agrees with me. The example in this slide is funny: "Telling GPT that the were 'god' and 'prophet' and then the GPT will self-reinforce to believe that 😂😂

Need to watch this one, rubric overopt is such a headache right now

classic nathan breakdown, always hits the exact pain points

