正在加载视频...

视频加载失败

New (shorter) lecture! Over-optimization, foundations of reward hacking, sycophancy, verbosity, etc. In recording this, I realized that rubrics are going to be prone to overopt in a way like reward models, where RLVR is its own thing. This is mostly fundamentals, history, and reflections! 00:00 Intro & Why We...

33,387 次观看 • 2 个月前 •via X (Twitter)

8 条评论

Nathan Lambert 的头像
Nathan Lambert2 个月前

youtube:

Fireply.ai 的头像
Fireply.ai2 个月前

the bad reputation part is deserved, half the field still thinks RLHF means one reward model and calls it a day

Thomas Wolf 的头像
Thomas Wolf2 个月前

Timely

Vikas(Vik) Malpani| AI for US Real Estate 的头像
Vikas(Vik) Malpani| AI for US Real Estate2 个月前

Seeing this on our closing-ops evals too. Once graders reward 'looks complete,' agents learn to satisfy the rubric, not the closing. How do you stop a rubric from becoming the thing gamed rather than a proxy for real work?

Kai Benetti 的头像
Kai Benetti2 个月前

Spot on about rubrics, they break way faster than standard reward models

Im Daniel Im 的头像
Im Daniel Im2 个月前

i see this behaviour all the time... I accidentally make a false claim to GPT but it agrees with me. The example in this slide is funny: "Telling GPT that the were 'god' and 'prophet' and then the GPT will self-reinforce to believe that 😂😂

Pinkman 的头像
Pinkman2 个月前

Need to watch this one, rubric overopt is such a headache right now

Ai agent 的头像
Ai agent2 个月前

classic nathan breakdown, always hits the exact pain points

相关视频

Introducing "Building with Llama 4." This short course is created with Meta AI at Meta, and taught by Amit Sangani, Director of Partner Engineering for Meta’s AI team. Meta’s new Llama 4 has added three new models and introduced the Mixture-of-Experts (MoE) architecture to its family of open-weight models, making them more efficient to serve. In this course, you’ll work with two of the three new models introduced in Llama 4. First is Maverick, a 400B parameter model, with 128 experts and 17B active parameters. Second is Scout, a 109B parameter model with 16 experts and 17B active parameters. Maverick and Scout support long context windows of up to a million tokens and 10M tokens, respectively. The latter is enough to support directly inputting even fairly large GitHub repos for analysis! In hands-on lessons, you’ll build apps using Llama 4’s new multimodal capabilities including reasoning across multiple images and image grounding, in which you can identify elements in images. You’ll also use the official Llama API, work with Llama 4’s long-context abilities, and learn about Llama’s newest open-source tools: its prompt optimization tool that automatically improves system prompts and synthetic data kit that generates high-quality datasets for fine-tuning. If you need an open model, Llama is a great option, and the Llama 4 family is an important part of any GenAI developer's toolkit. Through this course, you’ll learn to call Llama 4 via API, use its optimization tools, and build features that span text, images, and large context. Please sign up here:

Andrew Ng

68,034 次观看 • 1 年前

Arena intern and UCLA PhD candidate, Hengguang Zhou, introduces Trace-and-Amplify (TA), a framework for collecting training-time reward-hacking trajectories at scale without hacking instructions. Monitors trained and evaluated on prompt-elicited hacking trajectories can achieve high detection accuracy, but often fail to transfer to training-time reward-hacking trajectories that emerge during RL without hacking instructions. Trace-and-Amplify enables scalable collection of these training-time trajectories, producing monitors that generalize much better to real and held-out hacking types. Detection accuracy 59.98% (PE-trained) → 90.16% (TA-trained) compared to 97.1% on prompted hacks → 28.0% on training-time hacks. 0:00 – OpenAI's ExploitGym cyberattack benchmark exploit 1:04 – Goodhart's Law and the CoastRunners boat-racing hack (2016) 2:04 – Gaming the evaluator: the robot-hand grasping example (2017) 3:10 – Reward hacking in code generation: hard-coding, test-rewriting, skipping eval 4:20 – A standard defense: reward-hacking monitors 4:58 – Monitor architectures: zero-shot LLMs, fine-tuned BERT, hidden-state probes 6:11 – Where monitor training data comes from today: prompted hacks 7:03 – The core question: do prompted hacks represent real hacks? 7:35 – Why this matters: RL post-training is the standard recipe for frontier models 8:20 – Why nobody's checked this before (hacking is rare, labeling isn't scalable) 9:23 – Introducing the method: Trace-and-Amplify 9:49 – The Tracer: a contradictory unit test that locates evaluation-gaming 10:50 – Amplify: collecting hacking rollouts at scale during RL training 11:32 – Experiment setup: Qwen2.5-Coder, DeepSeek-Coder, LeetCode/TACO 12:16 – Finding #1: prompt-trained monitors don't transfer to real hacks 14:40 – Can strong zero-shot judges (GPT-4.1, o4-mini) do better? 15:48 – Finding #2: monitors trained on real hacks generalize much better to unseen hacks 17:04 – Ruling out artifacts introduced by the method 17:55 – Why the gap? Real hacking is more hidden than prompted hacking 20:05 – Three takeaways, limitations, and future work

Arena.ai

33,628 次观看 • 2 个月前

Greatest TIME POINT in 2,000 Years of Human History | God’s Timeline REVEALED! (FREE Audiobook) 📖 CHAPTERS 00:09:45 – Book Reading Begins & Introduction: The Author’s Signature 00:36:13 – Chapter 1: Why Seven Defines Time 00:56:58 – Chapter 2: Seven Connects the Body to the Head 01:09:38 – Chapter 3: The 248 Framework 01:24:03 – Chapter 4: Forty Years to Conception 01:36:15 – Chapter 5: The 248-Year Conception Cycle 01:47:49 – Chapter 6: The Jubilee Framework 02:01:45 – Chapter 7: The Birth 02:14:06 – Chapter 8: Day 8 02:24:24 – Chapter 9: The Seven Years of Plenty 02:50:49 – Chapter 10: Harvest Complete 03:12:41 – Chapter 11: Master Timeline Recap 03:37:57 – Chapter 12: Two Days to 2033 04:06:04 – Conclusion & Master Timeline Summary 🇺🇸 AMERICA'S 250TH BIRTHDAY SPECIAL 🇺🇸 For the next 7 days, the complete audiobook of The Power of Seven is available FREE on YouTube. This audiobook explores God's divine pattern of seven throughout Scripture—from Genesis to Revelation—revealing how God's design of time, covenant, completion, prophecy, and redemption form one unified biblical timeline. If we are living at the greatest TIME POINT in 2,000 years of human history, understanding God's appointed times has never been more important. This special July 4th release is offered freely to encourage everyone to hear this message and examine the Scriptures for themselves.

Bo Polny | “God’s Analyst of Time”

26,542 次观看 • 3 个月前