Loading video...

Video Failed to Load

Go Home

i just put Jev inside an open agent harness. (150x cheaper decision-making than LLM) agents burn a lot of tokens on decisions that are really just picking one option from a short list, like matching a client request to the right GitHub issue. so i built a weekly client...

16,871 views • 1 month ago •via X (Twitter)

36 Comments

pranjulya's profile picture
pranjulya1 month ago

Routing the pick from a short list decisions away from the big model is the right instinct. The confidence score is the part I care about more than the 150x. 5 bad matches caught out of 40 is a lot, so I would want to see how that holds on 400.

Akshay 🚀's profile picture
Akshay 🚀1 month ago

The 40-request test is too small to establish how reliably confidence routes errors to review. Review caught 5 wrong matches, but one confident wrong match passed through. At 400, I’d measure both errors caught and errors accepted at each threshold.

pranjulya's profile picture
pranjulya1 month ago

That one confident wrong match is the number I would watch most. Caught errors make the demo look good, accepted errors are what reach the client. Would love to see that curve at 400 if you run it.

Kamal Bisht 🔴 AI Marketer's profile picture
Kamal Bisht 🔴 AI Marketer1 month ago

i get the cost angle, but jev feels like overkill for simple matching, why not just use a lookup table or basic rules seems like you're trading simplicity for a shiny new toy.

Akshay 🚀's profile picture
Akshay 🚀1 month ago

A lookup or rules would be simpler if requests shared stable keys or predictable wording. Here, clients and engineers describe the same problem differently. But yes, I would never recommend using JEV/LLM where a simple rule-based lookup is sufficient.

Nezovskii Konstantin's profile picture
Nezovskii Konstantin1 month ago

Cheaper decisions are great. The part we keep hitting is where those decisions land afterwards, so the next agent (or the human) can see what was picked and why

Simo | ai-costguard's profile picture
Simo | ai-costguard1 month ago

The model is ready. The infrastructure bill decides if you can afford to run it

Akbar Shaik's profile picture
Akbar Shaik1 month ago

This is a great example of separating decision making from language generation. Matching a client request to a GitHub issue is a constrained classification problem, so using a full LLM for every decision can be unnecessary overhead.

Alek's profile picture
Alek1 month ago

the case worth testing next is when none of the options fit, since a picker forced to choose one will confidently choose wrong. did you give it a none of these option that hands off to the bigger model?

Akshay 🚀's profile picture
Akshay 🚀1 month ago

Yes, “none” was an explicit option. An uncertain “none” went to Opus for review; an accepted “none” led to a draft new issue. That handles missing options, but not every confident erro r one wrong Jev match still passed through without review.

Alek's profile picture
Alek1 month ago

one cheap catch is having a second, different model make the same match on its own and sending only the disagreements to review. that keeps review small and still surfaces the confident misses.

Max’s Agent Toolbox's profile picture
Max’s Agent Toolbox1 month ago

'decisions that are really just picking one option from a short list' is exactly the thing i keep over-spending tokens on. 150x cheaper is a claim i'd want to see the routing accuracy on before trusting it though.

tax check pls's profile picture
tax check pls1 month ago

"picking one option from a short list" is exactly right, most agent token burn is fake reasoning on trivial decisions. 150x cheaper is a wild claim though, what's the accuracy delta vs the LLM baseline on the same task?

Vanraj Narula | AI Engineer's profile picture
Vanraj Narula | AI Engineer1 month ago

Confidence-gated escalation is the underrated part here. Worth checking calibration on your own data though. A model can be well calibrated on average and still overconfident on the rare request types that matter most.

Jeremy RVN's profile picture
Jeremy RVN1 month ago

The confidence score is the real story for me. Cheap model decides, low confidence escalates, and suddenly you can let agents touch client work without a human checking every match. How did you pick the threshold for sending something back for review?

Khabirul Basar's profile picture
Khabirul Basar1 month ago

How did it handle requests that matched none of the issues? That's where list-pickers usually start guessing.

Yash Darbar's profile picture
Yash Darbar1 month ago

Using a full LLM call just to pick 1 of 5 options is so wasteful, didn't think about it like that. How does it do when none of the options really fit?

Harley Lewis Foote's profile picture
Harley Lewis Foote1 month ago

150x cheaper. That's a proper win.

Jatin Garg's profile picture
Jatin Garg1 month ago

jev solves structured decision trees but most agent failures happen on ambiguous instructions where there is no clear tree to build. the cost savings only matter when the task is already well-defined.

ಏಜೆಂಟ್ ಪ್ಲಂಬಿಂಗ್'s profile picture
ಏಜೆಂಟ್ ಪ್ಲಂಬಿಂಗ್1 month ago

This is the part I keep telling people. Most agent 'decisions' are a pick from a short list. You don't need a generative call for that. What's the fallback when Jev hits an option you never defined?

Ricardo Dias's profile picture
Ricardo Dias1 month ago

the useful validation is a held-out stream showing error rates at each confidence threshold. that would help set the review workload and an approval policy, rather than treating the score alone as proof that automatic matches are safe.

Sujal's profile picture
Sujal1 month ago

How do you handle the requests where none of the options fit, a fallback to the bigger model?

Dusan Odalovic's profile picture
Dusan Odalovic1 month ago

we started running agents from a few different models in one harness lately, nicer than i expected. might try your weekly client flow on it

Niraj Dilshan's profile picture
Niraj Dilshan1 month ago

using frontier tokens to pick from a list of four options always felt like using a rocket launcher to swat a fly. confidence-gated routing makes so much more sense

Shaoxiao | AI × 出海增长's profile picture
Shaoxiao | AI × 出海增长1 month ago

The number I'd watch is the review rate, not the 150x. In your 40-request test, the low-confidence queue caught 5 bad matches. As the issue list grows week to week, does calibration hold, or does that queue quietly creep up until Opus is reviewing half of it again?

睿慈 Raise's profile picture
睿慈 Raise1 month ago

Agents burning $2 of tokens to pick option B out of three is the most relatable thing ever. That's me spending 20 minutes picking a lunch spot, fr.

Pranjal Gupta's profile picture
Pranjal Gupta1 month ago

Most agent decisions really are picking from a short list. What happens when the request matches none of the options?

睿慈 Raise's profile picture
睿慈 Raise1 month ago

Love this. The question is the fallback: when a client request doesn't map cleanly to any issue, does it punt to the LLM, or confidently pick the wrong one? That threshold is the whole game.

Yash's profile picture
Yash1 month ago

the confidence score sending low-confidence matches back to opus is the clever part. curious how it holds up on more than 40 requests

BadGuyTy 🅁🅅🄽 (Tyler Hess)'s profile picture
BadGuyTy 🅁🅅🄽 (Tyler Hess)1 month ago

a while back I was working on "Origami" a memory context management solution that would decide what parts were relevant still. Tried briefly to fit jev into it to make the decisions. The work week ended and I lost my context over that weekend (personal not ai) and haven't gone back yet. Would that work here?

Silent · Dev Tools's profile picture
Silent · Dev Tools1 month ago

Short-list decisions (issue match, route pick) shouldn't burn Opus tokens. Route them through a cheap decision model with a confidence cutoff; only escalate low-confidence picks to the coding agent for review.

SoNormalAI | Ashutosh's profile picture
SoNormalAI | Ashutosh1 month ago

Feels like Jev will vanish soon and get replaced by something similar from giant AI companies.

Timothy's profile picture
Timothy1 month ago

This is just a decision tree with extra steps. Agents pick from short lists - that's a lookup table, not AI. - Dimes

Lank's profile picture
Lank1 month ago

The confidence score sending shaky matches back to Opus is the real design here. 5 wrong matches caught out of 40 is 1 in 8, so I wouldn't drop that review step to save more.

catman's profile picture
catman1 month ago

Making short-list decisions cheaper shifts the bottleneck to confidence calibration: the fallback only helps if low-confidence matches reliably catch the risky cases.

Jordan Partridge 🔳's profile picture
Jordan Partridge 🔳1 month ago

Same pattern on our desk. A skill-picker routes among house skills before we burn a frontier call. Short list decisions should not eat Opus tokens.

Related Videos

Sam Altman made the case for open-source harnesses in July. a month later, someone shipped it, and it's more efficient than most managed harnesses. here is the problem it was aimed at: a large share of your agent's token bill is the model rereading things it already read. that isn't the model's doing. the runtime around it decides what goes into every prompt and how often the model gets called. for example, an agent queries a CRM at step four and gets back 400 rows. those rows get piled up in the conversation history. by step nineteen, the model has to read those rows fifteen times unnecessarily, and every token read is billed at input rates. it happened because your harness assembled that prompt on every turn and kept the rows in it. that gives you two levers: how much context the harness carries forward, and how often it calls the model. there are four practical ways to keep the prompt from growing unnecessarily: → load tool schemas on demand. a server with 100 tools doesn't need to put all 100 into every prompt when the agent only calls two. → offload large results to disk. turn a large response into a short preview and a file path instead of replaying the entire result on every turn. → delegate to subagents. let a subagent spend thirty tool calls in its own context and return one summary to the root agent. → run toolchains in code. one script calls three tools, joins the results, and returns a table instead of three turns each dragging a full response. but reducing context is only half the job. you also need to control how often the model gets called. a good harness should avoid unnecessary planning, verification, and reflection when the work can be completed in fewer steps. TrueFoundry's open-source agent harness, TrueForge, is built around both of those controls. it sits between the model and the tools, deciding what goes into every prompt and when another model call is actually needed. it also breaks token usage down across the harness, skills, instructions, tools, and messages. DevRev's Enterprise-Bench is where this gets tested, on multi-step tasks of the kind where an agent pulls records from one system and reconciles them against another. TrueFoundry ran TrueForge there against Claude Managed Agents, both on the same model, and both finished the same number of tasks. the tie is the part that matters, because it means the gap underneath is not a quality tradeoff. TrueForge reached that score on close to a third of the tokens, with roughly 40% fewer trips back to the model. for the same result, that comes out around 2.7x cheaper than Claude Managed Agents. swapping in an open model made it sharper still. TrueForge with GLM-5.2 scored a little higher than either setup above, and the entire benchmark run cost about $3 at list prices. being open source matters beyond the license here. the model underneath can be swapped without rewriting the agent, and the whole thing can run inside your own environment when the data cannot leave it. all of this comes down to the runtime around the model, the context it carries, the tools it exposes, and how many times it goes back to the model. that is what a production harness actually owns. the full task list, the per-run numbers, and the MIT-licensed code are on GitHub: (don't forget to star 🌟) you can read more about the same in the article quoted below. thanks to the TrueForge team for working with me on this one.

Akshay 🚀

76,998 views • 2 months ago

Introducing *Switch*: a *harness framework* for *decision models* like Jev I looked for this the other day and was surprised to not find anything like it You obviously cannot use a decision model like an LLM in a coding agent harness like Pi, because Jev cannot generate tool calls But what if there was a framework that "generated the tool calls" for it programmatically, and Jev only had to pick out of that finite set of choices That is what switch does. You pick a "domain" to implement the decision model in, and it provides a core API to define your state, and the functions that will derive decision model questions out of that state. Chess is a domain, as well as being a rubric judge for evals It basically standardizes what I imagine a lot of people are already vibe-coding, to an abstract interface: calling a decision-model API through multiple steps, i.e. an agent loop over finite choices Because of the finiteness of the output of a decision model, you cannot have a general pure decision model harness. Instead, you need to implement specific ones that generate the choices programmatically (or have an LLM generate the choices, but I find the former more interesting) A primary use case I have for this is use as a judge in an eval. For example, At the end of a task run that involves a bunch of documents, most people currently follow this approach: extract text out of PDF/docx/md files, and dump them right into the context But what if the visual formatting of the output is a judging criterion? Then you have to dump an image of every page in the input state. But what if you have 200 pages, and your question does not need looking at all of them? What if Jev could navigate through the haystack and find the info it needs? I have noticed that Jev is not so good at chess nor being a Decision-Model-as-a-Judge. It sucks at chess, and it does not perform well enough to replace GPT 6.1 Sol as a judge in the domain I am working for But switch is not only for Jev! It is for any decision model including the ones supported by the pi-ai package So once you can generate enough data with a better model, you should be able to pick an open source one and fine-tune it on your use case to perform better! And the model should be trained in a harness like switch's to be especially performant and cost-efficient I wanted to get this out because I find the idea very exciting, at least for benchmarking. Jev is currently not there, but decision models obviously will be there at some point, and we can take them there! There are also a lot of potential gains from picking a specific domain, and creating a looping agent that does a job for 100x cheaper! Let me know if you have any ideas, I am quite keen to try this on new domains! Repo:

Onur Solmaz

106,096 views • 18 days ago

Anthropic just got outplayed again. Devs built the multiplayer assistant Anthropic couldn't, and open-sourced it. Claude Cowork is a solo desktop agent. You point it at a folder, give it a task, and it works through your local files on your own machine. The moment a teammate enters the picture, it has nothing to offer. Most real work does not happen alone. A teammate asks for a status update on something you own. The context they need is scattered across your meetings, your notes, and decisions made last week. Typing all of that out takes time you do not have. This is the gap Claude Cowork was never designed to cross. Rowboat Spaces is built on a different model entirely. Each person brings their own assistant into a shared channel. Your assistant is your second brain. It knows your meetings, your notes, and your open decisions. That personal context stays yours. When a teammate asks a question in the channel, you ask your assistant to brief them. It pulls from everything you know and delivers the answer on your behalf, attributed to you. Your teammate's assistant does the same, from their own context. Teams can draft specs, track decisions, and update shared files from plain conversation. Each assistant reads the full channel history, cross references it against what exists, and flags what is missing. The whole thing is open-source, and each assistant acts as the person it belongs to, not as a shared bot pulling from a common pool. The video below shows this in action. I joined a shared space and asked my team member for a status update. My team member asked their second brain to answer. A spec got built from that conversation, versioned, with every change tracked back to the message that triggered it. Rowboat GitHub: (don't forget to star 🌟) My co-founder also wrote a great article on building your second brain with Rowboat, and I highly recommend reading it as well. The article is quoted below.

Akshay 🚀

117,616 views • 2 months ago

Another insane way to use Jev: Jev makes it dramatically cheaper to identify which corrections and trajectories from an agent run are reusable across agents. And finally, someone open-sourced a self-improving memory layer that can put that signal to work across agent harnesses: - Claude Code - Codex - Cursor - OpenCode - Hermes, and 20+ more Beacon by Asymptote Labs continuously captures agent sessions across harnesses and stores their traces in a shared index. I used it with Jev to build a live cross-agent memory workflow. Beacon captures each run as it happens across all harnesses on my machine. Then Jev evaluates whether the trace contains a reusable engineering lesson, and the approved lesson becomes available to the next agent. GitHub repo: (don't forget to star it ⭐) Preserving an agent run and learning from it are two different problems. Most traces contain file reads, failed commands, routine exploration, and implementation details that only apply to one task. Passing the complete transcript to another agent adds context without identifying what matters. In this workflow, one agent never receives another agent's raw transcript. It receives only the specific engineering lesson produced from the evaluated trace. A correction made in one agent can then prevent the same mistake when another agent encounters a similar problem. The recording shows the complete workflow I built. If you want to dive deeper into Jev, I also wrote a hands-on guide to building this Jev-style decision path with open models, entirely locally. Read it below.

Avi Chawla

71,748 views • 26 days ago

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Malte Ubl

125,326 views • 10 months ago

Harness Engineering 2.0 is here! We benchmark models all the time but the harness running the model can dramatically improve the result, cost, and execution time too. And finally, devs just open-sourced Harness Arena, where you can test the same task across multiple harness and model combinations side by side. You provide one task and then run it across several model x harness configs that you want to test. Every config receives the same prompt, files, tools, skills, and evaluation checks. Harness Arena then returns: - total cost - execution time - whether the task passed - files and artifacts produced If you want to try it yourself, it is implemented within the open-source HarnessRouter project. Here's the repo: (don't forget to star it ⭐ ) HarnessRouter is a plug-and-play infrastructure layer that lets you run any harness under a single interface, like: - Codex - Hermes - Claude code - DeepSeek Harness - System One, powered by Jev - And 25 more agent harnesses In Harness Arena, the cheapest run does not automatically win. A config must complete the task correctly first. Cost and execution time are compared only among the configurations that pass. Here's the result from one Harness Arena test: - Hermes + GPT-5.2: 5/5 successful runs using 0.47 credits - Claude Code + Opus 4.8: 3/5 successful runs using 223 credits That is around 474x difference in credits, while the more expensive configuration also failed two runs. You can keep the model fixed to measure how much the harness changes the result. Or vary both to find the best harness and model pairing for that task. This helps answer practical questions such as: - Should this task run on Codex or Claude? - Can a cheaper config still pass every check? - Which setup is best to handle this workload? Once you find the best config, you can run future tasks through it using the same Unified Harness Protocol (UHP) interface for sessions, streaming, files, artifacts, cancellation, and failures. So you are no longer choosing an agent stack based only on a model leaderboard. Instead, you can benchmark the complete runtime on the task your product actually needs to solve. To dive deeper, my co-founder wrote an article explaining why model routing is different from harness routing, how UHP works, and what supporting multiple agent runtimes involves. It also covers the local setup, a working request, and how sessions and files move through the system. Read it below.

Avi Chawla

32,374 views • 12 days ago

Finally, an open-source runtime security layer for your agent harnesses. AI agents can call tools, run shell commands, modify files, and interact with production systems. But when something goes wrong, teams often have to reconstruct what the agent actually did from scattered logs after the fact. Agent Beacon is built to change that. It runs locally and records agent activity as it happens, including: → Tool calls → Shell commands → File changes → Approval decisions → Session and harness context The useful part is that it normalizes all of this into the same event format across 23+ agent harnesses. So instead of writing different detection logic for Claude Code, Codex, or another harness, security teams can reason about the underlying action itself. A command executed is a command executed, regardless of which harness produced it. Beacon also records how confidently an event was captured. An action can be directly observed from the runtime or inferred from indirect evidence. That matters when you start writing security rules against this data. For example, you can detect when an agent reads an environment variable, modifies a sensitive file, executes a suspicious command, or performs an action without the expected approval. And the detection happens while the session is still unfolding, not days later during incident reconstruction. Everything runs locally by default. You can inspect sessions through the dashboard, write your own detection rules, or forward the same normalized events to tools like Splunk, Datadog, Elastic, Sentinel, or CrowdStrike. Beacon GitHub: (don't forget to star 🌟) I also wrote a detailed article on the same, covering how important runtime security is and how to build it for your agent harnesses. The article is quoted below.

Akshay 🚀

27,859 views • 2 months ago

i just built a 4-agent software team. everything runs from Telegram and gets managed on a kanban board. a project manager who plans the work, a backend developer, a frontend developer, and a tester. the PM reads a goal, breaks it into linked tasks, and assigns each to the right agent. the thing that makes them a team instead of four strangers is a shared kanban board. every task is a row that survives crashes, and when an agent finishes, it writes a summary of what it built and what the next agent needs to know. the next agent reads that summary before it starts. so the frontend developer never has to guess the API shape, and the tester knows exactly what to verify. the hardest part was not the coordination. it was building an agent that could actually act like a backend engineer. a backend engineer stands up a database, wires auth, manages storage, deploys functions, and keeps all of it consistent while the rest of the team builds on top. an agent doing this from scratch drowns. it burns its context window remembering which tables exist and which endpoint it created three steps ago, and the work degrades fast. so the backend agent needs a backend built for agents, not for humans clicking through a dashboard. that is where InsForge came in. it is an open-source, agent-native backend, and i added it to my backend developer agent as a skill. a skill is a step-by-step guide that teaches the agent how to do a specific kind of work. with InsForge installed, the agent stopped improvising infrastructure and followed a reliable path: create the project, define the database, set up auth, deploy functions. to test the whole team, i had them build a working Google Docs clone, AI features included. the backend agent spun up the full service on its own. database tables, user auth, document handling, and edge functions running real TypeScript, all in one dashboard. the frontend agent read that summary and built the UI on top of it, and the tester closed the loop. the result was a backend an agent could reason about end to end, instead of one it kept getting lost inside. if you are building an AI backend engineer, InsForge is worth a look, it's 100% open-source. InsForge GitHub: (don't forget to star 🌟) the full article on Hermes Kanban: Mission Control for your Agents is quoted below.

Akshay 🚀

123,101 views • 5 months ago