Another insane Jev use case! Jev is making it dramatically cheaper to evaluate what actually happened inside an agent run. And finally, someone open-sourced a self-improving memory layer that can put that signal to work across agent harnesses: - Claude Code - Codex - Cursor - OpenCode, and 20+... more Beacon by Asymptote Labs continuously captures your agent history across harnesses and uses Jev to identify which runs are actually worth learning from. It then turns the highest-signal workflows, corrections, and debugging patterns into reusable skills. GitHub repo: (don’t forget to star it ⭐ ) Beacon preserves the complete session history. But preserving a run and learning from it are two different things. Most coding-agent sessions contain routine exploration, failed commands, and fixes that only apply to one task. The trace can remain available for inspection without turning every detail into guidance for future agents. Jev scores each run for evidence, reuse potential, and human correction signals. An application policy then decides whether to promote, review, or discard it. The recording shows this in action. Claude receives a coding task, modifies the implementation, and runs the tests. I then provide an edge-case correction, so Claude updates the code and adds regression coverage. Beacon automatically captures the complete session. Jev evaluates whether the correction contains a reusable engineering lesson. Once approved, that lesson becomes available to other coding agents working on the project. Since it works across harnesses: - Claude Code sessions can teach Codex. - Cursor debugging can improve OpenCode. So a problem solved by one agent should not need to be learned from scratch by another. If you want to dive deeper into Jev, I also wrote a hands-on guide to building this Jev-style decision path with open models, entirely locally. Read it below.show more

Incredibly excited about agent history becoming compounding knowledge. One breakthrough from your Claude Code session can become shared knowledge for Codex, Cursor, OpenCode, and every agent session that comes next. That’s the future we’re building toward with Beacon!

Thank you for the shoutout @_avichawla! Exciting times with Jev :)

the key is the promotion gate. agent memory gets worse fast if every successful command becomes doctrine. keep the full trace, but promote a lesson only when a human correction repeats or the pattern improves accepted work across tasks.

Yeah, and Jev is really good at making that decision.

Jev is not a cheaper GPT. It is a typed function call that returns a probability in ~100ms. I wrote up where it actually belongs in an agent stack and where the “can’t hallucinate” line falls apart.

Yeah. This might help understand how it differs from structured output decoding and regular LLM decoding:

Cross-harness learning is a great idea, but after Plugin4Shell I'd ask about the trust boundary. If a lesson from one agent becomes a skill for all the others, what stops a bad or poisoned lesson from spreading? Is the "approve" step always a human?

IMPORTANT: The evaluator should never write directly into an executable skill store. Human approval should remain mandatory whenever the promoted skill can modify code, credentials, or infrastructure.

self improving memory across tools is huge

最后这个“Cursor -”像是故意留的悬念

So many guides about Jev right now love it, appreciate for your work Avi

happy to help.

i'll qrt your tweet soon brother

Cool use case — Jev scoring which agent runs are worth learning from. On

Nice collection.

the cost for us wasn't the judging, it was access. ran a review over my own numbers this morning and it came back cannot verify on everything that needed the database, because its sandbox couldn't reach it. the one thing it caught was a word i couldn't defend.

The hard part of evaluating an agent run is usually the trace, not the eval logic. Browser/UI steps especially: screenshots and DOM snapshots are huge, so people end up sampling frames and losing the exact state that caused the failure. Curious whether Jev captures full action se

A tool event alone rarely reconstructs the failure. It is ideal to capture DOM plus a screenshot before and after each consequential action, then retain the full window around errors instead of sampling the whole run uniformly.

How does Jev score reuse potential?

Selecting which runs deserve promotion is the hard part. The score should reward fixes that survive a held-out task and penalize regressions on earlier workflows, or the memory layer will keep turning one-off recoveries into permanent rules.

I would evaluate the proposed rule, not rerun the exact trace. Generate a small test set around its claimed scope covering the original failure, nearby cases where it should apply, and counterexamples where it must stay as is. Promote where you have a positive regression delta.

It’s like putting an imbecile as a judge on top of a bunch of Einsteins. Perfect substrate for enterprise.

whats the actual cost gap against running an llm as the judge??

Working on this, will share insights pretty soon.

The challenge is defining what "actually happened" means when the agent took 47 steps across 3 tools. Are you checking final state, or whether it followed the intended reasoning path? Those are different evaluation targets.

I wouldn’t grade against one canonical reasoning path because several valid trajectories can reach the same state. In practice, I'd recommend deterministic checks for the final state, trace checks for tool or policy violations, and Jev only for whether the run contains a reusable lesson.

the idea of Claude teaching Codex is pretty cool. that could really streamline the learning process for agents

this applies not just across Claude and Codex, but 20+ harnesses since Beacon works across of them.

20+ harnesses is impressive. that really opens up a lot of possibilities for collaboration across models

This is the real problem: harnesses don’t share scars. AGENTS.md gets you partway. Shared docs plus a common memory store that captures from Claude Code to Pi to Grok Build is what's needed. Session logs that are never shared is archaeology, not shared knowledge.

Yes, AGENTS.md can distribute approved rules, but it does not capture where they came from or whether they still hold. A unified elementary is a really good solution over it.

Not every engineer has taste. Yet. A senior's Claude session beats a junior's because the taste is in the corrections, not the model. Capture those corrections as org memory, and every junior's agent starts coding like it passed through a senior's judgment.

Cheap eval of what happened in a run is usually the budget line teams cut. Pairing that with memory that works across Claude Code, Codex, and Cursor is the interesting bit.

Cross-harness is the key point — most memory tools lock you into one framework. How does Beacon handle conflicting memories from different agents: online updates during a session, or an offline consolidation pass?

now work this into llama.cpp new system one endpoint... put memory chunks back against the whole conversation that YOU ALREADY HAVE CACHED! Does this align or was it corrected Y/N? Done.

@_avichawla that's next level. evaluating agent runs was always a pain for me, this could save so much time.

hey Ali, lets chat!

the scorer deciding what's worth learning is itself unevaluated — jev's calibration numbers are self-reported. drift there and you keep the confident runs, not the good ones. 🥥

Why you need Jev when you have Ben

Beacon plus Jev is the self-improving memory layer everyone's been hand-waving about

The bit I'd want attached to every skill is when NOT to use it. A workaround for one repo can look like a universal rule once the original context is gone.

This is very cool!

A correction in Claude Code that Codex never sees is wasted work. The memory layer matters more than another model name.

The review step before a lesson becomes reusable is the part I like here. Keeping the original correction attached would help the next person judge whether that lesson still applies to their task.

The cross-harness piece is the key. Evaluation gets cheaper when the system can compare runs, not just replay logs. For teams tracking a customer journey across tools, that same lineage separates a bad decision from a broken handoff.

very cool!

eval is prob the best fit for jev so far tbh.. you want a fast yes/no on whether a step worked, not an essay about it. has anyone checked the calibration on those judgments against labelled runs yet?