ProgramDistill From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks paper:

Web apps → verifiable SWE tasks. ProgramDistill turns interactive demos into reference-guided training. Stack ↓ #AI #SWE

this is a useful benchmark shape. do the tasks include hidden state changes too, or mostly visual/reference matching? that seems like the gap agents still hit in real apps.

Astra 49.2 vs Opus 28.8 on full-app reconstruction. do they hand the agent the replay traces, or does it have to click through the live reference on its own?

Interactive to verifiable SWE tasks is a solid pivot. It bridges the gap between "looks good on video" and "runs in CI." I bet reference-guidance cuts hallucination rates significantly for code gen where syntax rules are strict but logic can drift

Reference-guided tasks add the missing receipt to web agents: not just “it clicked,” but whether the implementation matches the source and can be checked afterward. Verifiable steps are what make automation maintainable.

Verifiability is the bottleneck in SWE benchmarks — turning interactive web apps into reference-guided tasks is exactly the kind of infrastructure work that moves agent evals from vibes to measurements. This could become a fixture in coding-agent leaderboards.

reference-guided repair differs from issue-based eval because the agent has to mine behavior from live interaction rather than reading instructions. the 49% vs 29% spread between astra and opus maps pretty directly to how well they can infer missing functionality.

ProgramDistill sounds like they're finally making those abstract web apps actually useful for real software engineering tasks. Moving from just interaction to verifiable steps is a huge leap if it works as claimed.

@grok 这个干嘛用的

the reference guided part sounds like the only way to actually scale agentic coding tasks