正在加载视频...

视频加载失败

Before Fable got released (and pulled) Mozilla was quietly testing Claude Mythos against Firefox's 10M line codebase. The result? Over 400 security bugs fixes, including ones that had been hiding in the codebase for over a decade. Brian Grinstead, distinguished engineer at Mozilla, walked me through the agentic bug-finding...

260,557 次观看 • 4 个月前 •via X (Twitter)

31 条评论

claire vo 🖤 的头像
claire vo 🖤4 个月前

Ty ty to our amazing sponsors: 💼 @WorkOS - Make your app enterprise ready today 🏆 @MetaviewAI - the agentic recruiting platform for winning teams 🎤 Listen everywhere: ▶️ YouTube: 🎧 Spotify: 🍎 Apple:

claire vo 🖤 的头像
claire vo 🖤4 个月前

Full breakdown on the @chatprd blog:

Morgan 的头像
Morgan4 个月前

@hnshah @mozilla @bgrins Watching this for Brian’s loop pattern ideas!

Sajid AI 的头像
Sajid AI4 个月前

@mozilla @bgrins Insane reminder that most “AI breakthroughs” are really just better agent loops + verification, not magic models.

Fernando Moreira 的头像
Fernando Moreira4 个月前

This is exactly why deterministic enforcement matters more than ever. Even when agents find real value (like the 400+ bugs at Mozilla), the risk of them making destructive changes in production codebases is real — especially with frontend stacks using Cursor/Claude. I built Nemesis Defender for this: Pretool Hook (exit(2) before execution) + Iron Dome scanner + eBPF kernel protection. 100% local, tamper-proof, calibrated for Next.js/React/TS. Project + demo: Would love your thoughts on agent harnesses vs hard boundaries, Claire.

Rob | Magnanimous Craft | Custom Waves @ 1Heal.org 的头像
Rob | Magnanimous Craft | Custom Waves @ 1Heal.org4 个月前

@mozilla @bgrins friend developed a faster code search tool to make this easier called src-nav which does things grep cannot

Brooke Lacey 的头像
Brooke Lacey4 个月前

@mozilla @bgrins Ma’am I know I probs come off as a weird fangirl but these episodes are so informative and helpful - thank you for doing what you do! 😍

Sebastian Buzdugan 的头像
Sebastian Buzdugan4 个月前

@mozilla @bgrins finding 400 bugs is easier than proving the patches don't add regressions

Rafael Pinheiro 的头像
Rafael Pinheiro4 个月前

@mozilla @bgrins To point a model deterministically where duplications are in large codebases, dupehound does this really well, superfast and open-source.

Brosko 的头像
Brosko4 个月前

@mozilla @bgrins the 'you can build this in an afternoon' part is either the most motivating or most humbling thing ive read all week no in between tbh 😭

Abdulmuiz Adeyemo 的头像
Abdulmuiz Adeyemo4 个月前

@mozilla @bgrins Insane

AI Mastery Guide 的头像
AI Mastery Guide4 个月前

@mozilla @bgrins What's the "lie to the agent" part about, curious how that helped with false positives

Hidden Tape 的头像
Hidden Tape4 个月前

@mozilla @bgrins 400 fixes sounds great but id wanna know how many of those got properly reviewed before merge agent found it doesnt mean a human understood it

Sai Charan L 的头像
Sai Charan L4 个月前

@mozilla @bgrins The "50% setup" line is the whole lesson. Everyone argues about which model. The model is the cheap part now. The harness is the skill: a tight goal/loop and a verifier that kills false positives. Same model, two setups, completely different output.

Ralf Kronen 的头像
Ralf Kronen4 个月前

@mozilla @bgrins The "afternoon to write your own harness" line is the headline. If that harness plus a verifier yields 400 confirmed bugs, the result is a property of the setup, not of Mythos. Any capable model fits that slot. The scarce part was never the model, it was the scaffolding.

el_cuda 的头像
el_cuda4 个月前

@mozilla @bgrins I wanna know what lies to tell!

晚晚 的头像
晚晚4 个月前

@mozilla @bgrins 十年老 bug 被翻出来,这就有点狠了

Yossi Eliaz 的头像
Yossi Eliaz4 个月前

@mozilla @bgrins Yes, agent power stems from the model and harness, like Schumacher and a Ferrari.

Jens Honack 的头像
Jens Honack4 个月前

@mozilla @bgrins Mythos was able to find some really gnarly bugs too. Impressive considering Firefox has some extremely bright engineers. As competent the human mind is, it's simply not as good a repository of structured information, as LLMs, which is probably why LLMs excel at this.

Simply Complex Systems 的头像
Simply Complex Systems4 个月前

@mozilla @bgrins The only gates to a deeply integrated, self directed agentic environment are careful reading and well articulated questions.

Picassio 的头像
Picassio4 个月前

@mozilla @bgrins lying to the agent to get better results is the part im stuck on that sounds wild and i need the whole explanation NOW

Gregor 的头像
Gregor4 个月前

@mozilla @bgrins not sure 400 fixes is the headline here, curious what the validation loop actually looked like. at 10M lines, how many of those patches had a human review before merge?

Pawzard 的头像
Pawzard4 个月前

@mozilla @bgrins decade old bugs just vibing in the codebase like a tenant who stopped paying rent in 2014 but nobody checked 💀

★ Adam LG Ring ★ BSc - IBM 的头像
★ Adam LG Ring ★ BSc - IBM4 个月前

@mozilla @bgrins the load on newly released models is going to be insane from now on who knows how long it will be up !!!

A 的头像
A4 个月前

@mozilla @bgrins the model isn't the product. the loop around the model is where the leverage hides.

David Jones 的头像
David Jones4 个月前

In the agentic security bug pipeline for Firefox, how did you handle prioritizing and sequencing work across the codebase (or potential issues), respecting compute costs, dependencies, or human review bandwidth — and did the harness integrate directly with existing bug tracking/backlog tools, or was that kept fully human-driven?

Axel Winter 的头像
Axel Winter4 个月前

@mozilla @bgrins 499 isnt much though

Benign Rad-Science 的头像
Benign Rad-Science4 个月前

@mozilla @bgrins Don’t you think it’s wierd that Anthropic have it to Mozilla for a browser rather than to the US govern to secure its systems? Wierd.

Prakhar 的头像
Prakhar4 个月前

@mozilla @bgrins I feel this is like tech debt and you have to get to it sooner or later. But agents make it possible to do so fast and well, that it is potentially not as big of a deal as before. I see companies releasing their stacks in rust so frequently now.

Adel Bucetta 的头像
Adel Bucetta4 个月前

@mozilla @bgrins this is massive. the real unlock isn't just testing your model against a big codebase, it's what that says about the state of code maintenance and how we need tools like claude to catch up with our own sloppiness

Makaroni 的头像
Makaroni4 个月前

@mozilla @bgrins the 50/50 split is the real takeaway people keep crediting the model but the harness doing half the work is not a small detail

相关视频

Karpathy said something you'll regret ignoring: "We have to keep the AI on the leash. I'm still the bottleneck. I have to make sure this thing isn't introducing bugs and that there's no security issues." He said it at YC talk last year, when the worry was reliability. The models hallucinated and made mistakes no human would, so the leash implied keeping yourself in the loop and checking the output before trusting it. The models are far better now, and the line still holds, for a reason he was not focused on back then. Even a model that writes flawless code today still has no idea who is allowed to run it. Correctness and authorization are different problems, and only correctness improves as the model improves. A perfect agent still hands a tool where anyone can do anything, because permission was never part of the task. I actually tested this in practice with Claude Code. I asked it to build a small internal tool with a button that issues account credits. It worked first try, and running it locally, the credit applied the instant I clicked. Nothing decided who was allowed to click it. The agent wrote the right logic and displayed a success notification. It never checked whether the caller had the right, whether it should pause for a human, or whether anything was logged. And this is not a bug a smarter model can outgrow because the leash was never in the code. Identity, permissions, and audit live in the system that runs the app, not in what the agent generates. To solve this, I took the exact same bundle and hosted it on Retool. The credit write that fired silently on my laptop now stopped at an approval gate, resolved to a real identity through SSO, and landed in an audit log. I wrote none of it. The app inherited the entire boundary the moment it was deployed, and the video shows the before and after. You can try it yourself here: I also wrote a detailed breakdown of the whole thing in my recent article, and I worked with the team to put this together. It walks through the build, the exact moment the credit write went through on my laptop with nobody checking, and then what changed when the same app ran on Retool. It also covers why this is a property of the runtime and not something a better model fixes, which is why devs typically miss this. The article is quoted below.

Akshay 🚀

42,911 次观看 • 4 个月前