Melissa Pan's banner

Melissa Pan

@melissapan • 5,229 subscribers

CS PhD @UCBerkeley Sky Lab 🐻 Systems & AI & Sustainability 🌍 Prev: @google, @ibm, @CarnegieMellon🐕‍🦺, @UofT🇨🇦

Shorts

Finding 1: Harness affects cost more than correctness. Fable 5 can cost twice as much for a 1.1-point gain in success rate. On SWE-bench Lite: - Claude Code: 97.8% accuracy, $1.33/rollout - Pi: 96.7% accuracy, $0.67/rollout While Claude Code reaches the highest success rate on the SWE-bench Lite frontier, Pi and Codex often achieve similar success rates at lower cost across the models we test. So you may be paying a hidden “harness tax” if you pick your harness based only on the success rate… 💸 (2/n)

Finding 1: Harness affects cost more than correctness. Fable 5 can cost twice as much for a 1.1-point gain in success rate. On SWE-bench Lite: - Claude Code: 97.8% accuracy, $1.33/rollout - Pi: 96.7% accuracy, $0.67/rollout While Claude Code reaches the highest success rate on the SWE-bench Lite frontier, Pi and Codex often achieve similar success rates at lower cost across the models we test. So you may be paying a hidden “harness tax” if you pick your harness based only on the success rate… 💸 (2/n)

14,563 просмотров