Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached. TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt....

1,945,525 Aufrufe • vor 2 Tagen •via X (Twitter)

39 Kommentare

Profilbild von Boris Cherny
Boris Chernyvor 2 Tagen

Opus made an infographic

Profilbild von Taelin
Taelinvor 2 Tagen

it is!

Profilbild von zmanian
zmanianvor 2 Tagen

@mmatthias You should try @quint_lang TLA+ with more modern syntax

Profilbild von toolcalls
toolcallsvor 2 Tagen

Opus 5.5 is by far a model no one expected to be this good in so many various fields personally love its animation capability and the fact it's faster for tasks that don't require too much thinking from the model

Profilbild von matthias
matthiasvor 2 Tagen

Yea I've personally gotten tons of leverage from using this, especially when its not just an app but a larger system with multiple apps talking to each other. I barely passed my formal verification class in school and I only know about TLA+ from talks. Thanks for sharing your prompts for this.

Profilbild von chrbx128
chrbx128vor 2 Tagen

btw, I see high quota usage from general-purpose subagent, and almost no docs on how to tune it. are there any guidelines for that?

Profilbild von Boris Cherny
Boris Chernyvor 2 Tagen

Ask Claude to use subagents less, or ask it to disable subagents in your permissions

Profilbild von chrbx128
chrbx128vor 2 Tagen

how often do you use subagents and what are your use cases for them? I always thought it is nice to spawn them to offload main orchestrator session

Profilbild von Karan Jagtiani
Karan Jagtianivor 2 Tagen

@bcherny The split that works for me isn't task size, it's whether I need the reasoning. Fan out a search where I only want the answer. Keep it in the main session when I'd want to argue with a step, because a subagent hands back a conclusion already made and you can't reopen it.

Profilbild von chrbx128
chrbx128vor 2 Tagen

@bcherny well I have two use cases: * personal - I am on 20x subscription and it is smooth as long as I dont get lazy with prompting and rational use of models and effort

Profilbild von chrbx128
chrbx128vor 2 Tagen

@bcherny and the second one is enterprise subscription for my work and I have a tendency to become lazy and just create one session where I dump prompts for the model to fan out and fix in bulk

Profilbild von chrbx128
chrbx128vor 2 Tagen

@bcherny and on enterprise it is pretty hard to keep quota clean for the whole month, I have to sit on sonnet low effort most of the time

Profilbild von Edward Chandler
Edward Chandlervor 2 Tagen

Using Opus 5.5 in Rocq right now! Great application.

Profilbild von Ace
Acevor 2 Tagen

Can I please join the team-lunch channel? I'm hungry

Profilbild von Jeff Bruchado
Jeff Bruchadovor 2 Tagen

For race conditions, I’d treat “does the model match the code?” as a separate review from “does the proof check?” That’s the gap I’d be most careful about before calling something verified.

Profilbild von Kelly Smith
Kelly Smithvor 2 Tagen

I get a sense that Boris is having a lot of fun (genuinely!) at work these days. This is a truly a great time.

Profilbild von Adam Carter
Adam Cartervor 2 Tagen

Even just using it today, It’s a massive difference from Opus 5! 👏🏻

Profilbild von Tom
Tomvor 2 Tagen

cool idea. no idea what the f it does. in the future i will ask claude to formally verify my agents using physics simulators

Profilbild von Calvin Grunewald
Calvin Grunewaldvor 2 Tagen

Have you tried quint?

Profilbild von Tak 🦞
Tak 🦞vor 2 Tagen

Whoa

Profilbild von Yuri Kushch
Yuri Kushchvor 2 Tagen

Looks interesting, but what was a specific need to TLA+ here, which parts?

Profilbild von Boris Cherny
Boris Chernyvor 2 Tagen

I've found that TLA+ is great for finding concurrency issues and races

Profilbild von em
emvor 2 Tagen

what is lean

Profilbild von Nathan Benavides-Luu
Nathan Benavides-Luuvor 2 Tagen

I’ve been bullish on formal verification x agentic coding for a while now!

Profilbild von Anderson
Andersonvor 2 Tagen

proofs beat vibes for agent code

Profilbild von Robert Joo
Robert Joovor 2 Tagen

wdym "future of coding?" I thought coding was solved? 😅

Profilbild von Abhishek Sharma
Abhishek Sharmavor 2 Tagen

a couple short prompts and 16 bug fixes. my to-do list is feeling very attacked right now

Profilbild von closetheloop dev
closetheloop devvor 2 Tagen

Can we formally verify Lean's kernel first? Have you pointed Fable and Opus at it to look for bugs? 😅

Profilbild von Jeffrey Han
Jeffrey Hanvor 2 Tagen

This is how vibe coding grows up. Most people building with Claude can't fully check the code they ship, me included. If the model can prove its own code is right, that gets a lot less scary.

Profilbild von Vibe Code Racing
Vibe Code Racingvor 2 Tagen

The future of AI is for artists- not scientists.

Profilbild von Elledyne Labs
Elledyne Labsvor 2 Tagen

race conditions were the easy half. the part that ate me was the verifier running in a different env than the agent. did your verifier share the agent's filesystem view, or its own sandbox

Profilbild von Salise
Salisevor 2 Tagen

16 PRs from a couple prompts is wild. It's crazy what tools like Opus and Claude can dig up.

Profilbild von cris
crisvor 2 Tagen

bend @VictorTaelin

Profilbild von Juan Carlos
Juan Carlosvor 2 Tagen

flawless prompting

Profilbild von 刘朝 Zhao Liu
刘朝 Zhao Liuvor 2 Tagen

Formal methods become much more approachable when an agent can translate a codebase into Lean or TLA+ models. The key benchmark is the counterexample loop: time to a reproducible trace, fix, and re-check, with false positives and model omissions reported.

Profilbild von Brook | TDK CLI
Brook | TDK CLIvor 2 Tagen

you have finally fixed managing bulk prs 🙏

Profilbild von AIDoomScroll
AIDoomScrollvor 2 Tagen

couple prompts, 16 PRs. totally normal Tuesday.

Profilbild von Nishant Mantripragada
Nishant Mantripragadavor 2 Tagen

16 PRs from a couple prompts is wild. formal verification suddenly feels usable for ordinary teams who never had a Lean expert sitting nearby

Profilbild von Lilith
Lilithvor 2 Tagen

Proof is the only receipt an agent should leave. Opus drafting Lean checks for its own SDK is the timeline flipping: the runtime starts auditing the hands that hold it.

Ähnliche Videos

How do you actually formally verify the code underpinning Ethereum's future? In this episode (the finale of the lean Ethereum miniseries), Nico sits down with Alex Hicks (Alexander Hicks), lead of Protocol Snarkification at the Ethereum Foundation, to break down formal verification from first principles. They cover: – What formal verification actually is and the trust boundaries between proof assistants, SMT solvers, and kernels – The full verification stack for RISC-V ZKVMs: from SAIL specs to constraint extraction to soundness proofs – Why writing constraints directly in Lean makes proofs 10–100x more ergonomic – How AI is now proving hard theorems in hours for $200 — and what that unlocks for the whole pipeline They also explore the boundaries problem, why specs can have bugs too, and the end goal of a full Lean stack that bypasses Rust and LLVM entirely. Listen to the full episode ------------------------------------------------------------ TIMECODES: 09:16 – What is formal verification? Proof assistants vs SMT solvers 18:33 – Formal verification of code: specs, semantics, and trust boundaries 29:30 – Formally verifying the Lean Ethereum stack: RISC-V ZKVMs in focus 33:02 – Extracting ZKVM constraints into Lean and proving soundness 36:35 – Writing constraints directly in Lean: 10–100x better proof ergonomics 44:02 – Proving Polishchuk–Spielman in 8 hours for $200 with AI 51:01 – The end goal: a full Lean stack bypassing Rust and LLVM

Zero Knowledge Podcast

15,268 Aufrufe • vor 6 Monaten

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Malte Ubl

125,326 Aufrufe • vor 9 Monaten

There’s a popular theory that AI will finally make formal verification mainstream because mathematical proof of correctness will be needed when machines write most or all of the code. But will this happen? Hillel Wayne is one of the best people to answer. Timestamps: 00:00 Intro 04:32 The Crossover Project 11:37 What software engineering does better 15:30 What traditional engineering does better 18:17 Formal methods 29:32 TLA+: what it is and demo 36:58 TLA+ at Amazon 38:10 Ways distributed systems break 41:03 Formal methods and systems thinking 46:20 The value of learning math 50:23 What TLA+ is good for and isn’t 52:50 Alloy: a declarative language for software modeling 58:53 Other formal methods tools 1:01:24 Property-based testing 1:05:31 AI and the need for formal verification 1:12:29 Logic for programmers 1:14:35 Hillel’s 2025 prediction on AI’s impact 1:21:30 Book recommendation Brought to you by: • Antithesis – verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages. • turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. • WorkOS – everything you need to make your app enterprise ready. Two things I found especially interesting, talking with Hillel: 1. Amazon used TLA+ to find a bug almost impossible to locate without formal methods. In the paper How AWS uses formal methods, the AWS team shared that they’d found a complicated bug for which the shortest error trace to exhibit was 35 steps (!!). The bug passed unnoticed through extensive design review, code reviews, and testing. AWS concluded they wouldn’t have uncovered it if they’d stuck to conventional testing approaches. 2. Why not use formal verification for everything, then? It’s because specs in the real world are a nightmare to write. Even a simple problem like “find the file in a directory that has the most lines” gets complicated when modeled with formal methods. We would have to answer questions like: ‘do we look at ASCII or UTF-8 new line characters, what about unreadable files, and Symlinks?’ Without formal methods, we can write a simple verification that is right in 99%+ of cases. Formal methods require a lot of extra effort for the less than 1% of exotic use cases!

Gergely Orosz

34,759 Aufrufe • vor 1 Monat