Akshay ๐
@akshay_pachaar โข 290,670 subscribers
Simplifying LLMs, AI Agents, RAG, and Machine Learning for you! โข Co-founder @dailydoseofds_โข BITS Pilani โข 3 Patents โข ex-AI Engineer @ LightningAI
Shorts
Videos
Finally, an OpenRouter for agent harnesses! (including System One by Jev) Devs just open-sourced a plug-and-play infrastructure layer that lets you run any harness under a single interface, like: - Codex - Hermes - Claude code - DeepSeek Harness - System One, powered by Jev - And 9 more agent harnesses This means you can bring Jev into the same product that already uses Codex, Claude Code, or another supported harness, without writing another implementation for sessions, streaming, files, cancellation, and failure handling. Here's the repo: (don't forget to star it โญ ) The harnesses run locally, and the Unified Harness Protocol (UHP) defines the common task interface with an OpenAI Responses-compatible API. If you want to dive deeper, my recent article explains why model routing is not the same as harness routing, and what it takes to support multiple harnesses. It also covers UHP, the full local setup, a working API call, and how sessions and files work. Read it below.
Akshay ๐359,592 Aufrufe โข vor 26 Tagen
This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time. โ llama.cpp reached 22.0 tokens/s โ MLX reached 25.1 tokens/s โ Uzu reached 92.1 tokens/s The video shows all three runs side by side. The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding. Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory. A base M5 provides about 153 GB/s of memory bandwidth. At one token per pass, the theoretical ceiling is roughly 153 รท 5.2, or 29 tokens/s, however fast the GPU computes. Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon. Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass. Uzu pushes this further in two ways. โ It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences. โ It verifies several possible sequences as a tree, then commits state only for the path the full model accepts. The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass. In my run, every pass of the 9B model produced 7.5 output tokens on average. Ordinary decoding produces exactly 1. That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29. The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPUโs accelerated int8 matrix path. It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript. The repository is here: (don't forget to star ๐) I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code. The article is quoted below.
Akshay ๐80,238 Aufrufe โข vor 8 Tagen
Chinese researchers did it again! OpenBMB just open-sourced MiniCPM5-2B, a dense 2B-parameter model built for reasoning, coding, and tool use on resource-constrained hardware. Artificial Analysis ranked it highest among models under 4B in its Agentic Index comparison. It scored 20, while Granite 4.2 8B scored 9. The model is particularly strong at coding and tool calling, so I tested both capabilities locally. I pulled it onto my machine, connected it to a constrained CI repair agent, and gave it one issue: > A customer reports that retrying checkout with the same idempotency key returns a larger total. The first request returns $109, while the retry returns $118. Find the root cause, fix it without changing the public API contract, and verify the complete test suite. The Python checkout service had 18 tests. Sixteen passed, while two failed on the retry path. The agent could list files, search code, read selected ranges, run approved tests, apply a patch, and inspect its diff. It reproduced the failure, then followed the checkout and idempotency paths through the repository. The model found that shipping was added to mutable order state before the cached result was checked. On retry, the same order already contained shipping, so the calculation added it again. It generated a narrow patch that moved the idempotency check ahead of the mutation without changing the public API. The agent ran the targeted tests and the complete suite. All 18 tests passed. The model was never told which file contained the issue or what change to make. Each test result, search result, and code inspection determined its next action. The video below shows the full trajectory, including the investigation, tool calls, generated patch, diff, and final verification. Everything ran 100% locally on my machine throughout the run. MiniCPM5-2B supports llama.cpp, Ollama, vLLM, SGLang, iOS, Android, and HarmonyOS for local deployment. The model weights, training recipes, reasoning datasets, and UltraX data-refinement system are open-source. GitHub Repo: A 2B model can now inspect a repository, reason across multiple files, modify code, and verify its patch while remaining small enough to target local hardware.
Akshay ๐313,956 Aufrufe โข vor 1 Monat
I just trained my own Jev-style Decision model locally! With Qwen3.5 0.8B as the base, accuracy jumped from 37% to 65% in just 60 steps (10 minutes of training). All on just 4GB of VRAM. This video shows how you can do the same. Chapters: 0:00 - Intro 0:50 - Set up Unsloth Studio 3:47 - Pick a model and dataset 5:19 - Train with LoRA 7:51 - Test it in the app 10:03 - Same workflow in code 12:38 - Train and evaluate (37% โ 65%) 14:46 - Save and run inference 15:24 - Outro If youโre new to JEV and want to understand how decision models work, Iโve written a detailed article about it. Itโs quoted below.
Akshay ๐57,885 Aufrufe โข vor 7 Tagen
NVIDIA and Stanford just challenged Jev. (their new System 1 architecture runs up to 9x faster.) It is called a Contrastive Language Model, or CLM. Like Jev, CLM is not designed to generate text. It handles the small, repeated decisions inside AI systems, such as choosing a tool, ranking a patch, routing a request, or selecting the next action. But CLM reaches those decisions differently. Instead of generating an answer token by token, it treats decision-making as a retrieval problem. Here is how it works. 1) Encode the state CLM takes the current situation, such as an agentโs context or the state of a game, and converts it into a vector. It uses a frozen Qwen3-8B model with a small trainable state projection head. 2) Encode every possible action A separate action head converts each candidate into the same vector space. In the Mario example, the candidates are left, jump, and right run. CLM does not invent a fourth option. It only evaluates the actions supplied by the application. 3) Learn which states and actions belong together During training, the correct state-action pair is pulled closer while incorrect pairs are pushed apart. A batch of B examples produces a B ร B similarity matrix. The matching pairs sit on the diagonal. Every other pairing becomes a negative example. This contrastive training uses InfoNCE, the same general mechanism behind systems such as CLIP and dense retrieval. 4) Turn similarity into a decision At inference, CLM measures the cosine similarity between the state and every candidate action. A softmax converts those scores into a probability distribution. The application can choose the winner, apply a confidence threshold, or escalate an uncertain result. The real speed advantage comes from separating states and actions. Actions can be embedded once and cached. If an agent repeatedly chooses between the same tools, CLM only needs to encode the changing state and compare it with stored action vectors. That replaces repeated generation with one embedding pass and a set of cheap dot products. The researchers report that CLM-8B matches Jev across computer-use, gaming, and tool-calling evaluations while reaching up to 9x lower latency. The improvement is largest when actions repeat or the candidate set grows. CLM still has limits. It cannot generate new actions, its probabilities are relative to the supplied candidates, and its strongest verifier results require task-specific fine-tuning. But its central idea is powerful. The entire research is open-source, including the code. Read more here: When software already knows the possible answers, an AI model should score them instead of generating more words. I also wrote a full breakdown on how system one models like Jev work. The article is quoted below.
Akshay ๐164,126 Aufrufe โข vor 22 Tagen
Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. GitHub repo: Talking about RAG, I recently wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x. The article is quoted below.
Akshay ๐949,581 Aufrufe โข vor 3 Monaten
UC Berkeley just open-sourced FreeToken. (2โ4x faster local LLM inference than Ollama) the results are wild: - Qwen3.6-35B on an 8GB GPU at 39.3 tokens/s - DeepSeek-V4-Flash 284B on a 32GB GPU at 22 tokens/s - GLM-5.2 753B on a 96GB GPU at 14.9 tokens/s a 35B model at 16-bit precision needs about 70GB just for its weights. even at 4 bits it is close to 18GB, and FreeToken serves it on an 8GB GPU. let me explain how: all three models mentioned above are Mixture-of-Experts, and that is what FreeToken takes advantage of. each layer holds hundreds of separate experts plus a small router that picks a few of them per token. Qwen3.6-35B activates roughly 3B of its 35B parameters per token. DeepSeek-V4-Flash picks 6 of 256 experts per layer, so 13B of its 284B run at a time. so compute was never the bottleneck. the weights a single step touches fit comfortably on a consumer GPU. every expert the router might pick still has to exist somewhere. they sit in system RAM, and the GPU keeps a cache of the ones the model has been using recently. so everything comes down to what happens when the router picks an expert that is not on the GPU. there are two ways to serve that miss: 1. copy it over PCIe and run it on the GPU 2. run it on the CPU, where it already lives both read from the same system memory, so they compete for one pool of bandwidth instead of adding to each other. existing engines pick one option and freeze it when the model loads. but routing changes on every token, so a fixed choice misses most of what the model asks for. FreeToken measures both bandwidths on your machine and splits each step's misses between the two paths in proportion. the GPU and CPU results then merge exactly, with no approximation. two machines with the same GPU can end up wanting opposite strategies, which I did not expect. a 5090 in a gaming desktop should push nearly everything over PCIe, while an 8GB laptop is better off computing most misses on the CPU. none of that is readable off a spec sheet, so the engine profiles it once per machine. the second half of the design is about agents. coding agents constantly rewrite their own history, and every edit normally forces thousands of tokens back through prefill. FreeToken saves its checkpoints at the exact boundaries agent frameworks cut on, so it only reprocesses the new part. its slowest first token stays under 44 seconds, while llama.cpp peaks at 232 and KTransformers at 946. it serves the OpenAI and Anthropic APIs under Apache 2.0, so Claude Code and Codex can point at it directly. releasing weights publicly decides who can download a model, not who can afford to run one. frontier open models keep shipping, and running them still assumes a rented cluster. meanwhile there are over a hundred million consumer machines with discrete GPUs sitting mostly idle. closing that gap was never a hardware problem, and work like this is what turns open weights into something you can actually use. paper: repo: almost every idea in this post, from why memory bandwidth decides the outcome to why moving weights costs more than computing on them, comes straight out of how a GPU is built. I wrote a detailed primer on that. the article is quoted below.
Akshay ๐345,042 Aufrufe โข vor 1 Monat
Another insane Jev use case! Jev makes it incredibly cheap to evaluate and classify agent runs at scale. And finally, someone open-sourced a self-improving memory layer that can put that capability to work across agent harnesses. It turns your agent sessions into a compounding knowledge layer, where every successful run can make future agents smarter across: - Codex - Claude Code - Cursor - OpenCode and 20+ more Beacon by Asymptote Labs continuously builds a shared history across your agent harnesses and uses Jev to identify the runs worth learning from. It then turns the best workflows, corrections, and debugging patterns into reusable skills. GitHub repo: (donโt forget to star it โญ) Most agent runs are messy. They contain exploration, failed commands, dead ends, and one-off fixes that should never become permanent memory. So Beacon preserves the full session history, while Jev helps decide what should be promoted, reviewed, or discarded. The recording below shows this in action. Beacon found 579 sessions across 5 coding-agent harnesses and normalized them into one consistent history. From there, Jev surfaces the lessons worth keeping and makes them available across your agent stack. - A pattern learned in Cursor can carry into OpenCode. - A lesson from Claude Code can improve the next Codex run. Every successful run adds to the shared knowledge layer, making future agents smarter. If you want to dive deeper into Jev, I also wrote a breakdown of how it works. The article is quoted below.
Akshay ๐120,298 Aufrufe โข vor 23 Tagen
everyone is sleeping on this new JSON extraction model! - open-weights - matches Gemini Flash 3.5 - while being 3x faster pulling structured data out of documents usually takes a chain of OCR, an LLM, and extra code to repair broken JSON. lift, from Datalab, does all of this in a single step. you give lift a PDF or image along with a JSON schema, and get back a JSON object that matches that schema. the model reads every page of a document in one pass, so values that span multiple pages still come out right. any document you can describe in a schema works, from invoices to research papers. here's what you get with lift: - 90.2% field accuracy on 11,000 fields - 9B model, nearly matches Gemini Flash 3.5 at 3x the speed - built not to hallucinate; returns null if missing - self-hostable with vLLM or Hugging Face GitHub repo: (don't forget to star ๐)
Akshay ๐60,661 Aufrufe โข vor 14 Tagen
Anthropic just got outplayed again. Devs built the multiplayer assistant Anthropic couldn't, and open-sourced it. Claude Cowork is a solo desktop agent. You point it at a folder, give it a task, and it works through your local files on your own machine. The moment a teammate enters the picture, it has nothing to offer. Most real work does not happen alone. A teammate asks for a status update on something you own. The context they need is scattered across your meetings, your notes, and decisions made last week. Typing all of that out takes time you do not have. This is the gap Claude Cowork was never designed to cross. Rowboat Spaces is built on a different model entirely. Each person brings their own assistant into a shared channel. Your assistant is your second brain. It knows your meetings, your notes, and your open decisions. That personal context stays yours. When a teammate asks a question in the channel, you ask your assistant to brief them. It pulls from everything you know and delivers the answer on your behalf, attributed to you. Your teammate's assistant does the same, from their own context. Teams can draft specs, track decisions, and update shared files from plain conversation. Each assistant reads the full channel history, cross references it against what exists, and flags what is missing. The whole thing is open-source, and each assistant acts as the person it belongs to, not as a shared bot pulling from a common pool. The video below shows this in action. I joined a shared space and asked my team member for a status update. My team member asked their second brain to answer. A spec got built from that conversation, versioned, with every change tracked back to the message that triggered it. Rowboat GitHub: (don't forget to star ๐) My co-founder also wrote a great article on building your second brain with Rowboat, and I highly recommend reading it as well. The article is quoted below.
Akshay ๐117,616 Aufrufe โข vor 1 Monat
Redis built a cache that cuts LLM costs by 70%! Production LLM apps often receive different versions of the same question. For instance, an internal developer assistant might receive: - "How do I rotate an API key?" - "Where can I replace my API key?" The wording is different, but both questions have the same answer. Prefix caching cannot handle this repeated generation because it only reuses computation when the cache matches bit-by-bit. And if the cache hits, an LLM response is still generated again. A semantic cache stores the complete question-response pair outside the model. When a query arrives, it searches for previously answered questions with similar meaning. A valid match returns the stored response with no LLM call. If you want to use this in practice, Redis already implements it as a managed service called Redis LangCache. Under the hood, LangCache embeds the incoming question, searches stored responses, and applies the configured similarity threshold and filters. A cache hit returns the earlier response. A miss falls back to the LLM, after which the new response can be stored for future requests. Redis also handles access scopes, custom filters, embedding selection, TTL, eviction, and monitoring through a REST API, without another database to deploy or manage. While the actual savings depend on how much safe repetition exists in the workload, cache-hit responses are up to 15x faster and 70% cheaper. Try Redis LangCache: I built a small interface comparing LangCache with direct LLM inference. The video below shows this in action, and I worked with Redis on this post to put it together. To dive deeper, my co-founder published a detailed article on KV, prefix, prompt, and semantic caching. Read it below.
Akshay ๐88,731 Aufrufe โข vor 24 Tagen
i just put Jev inside an open agent harness. (150x cheaper decision-making than LLM) agents burn a lot of tokens on decisions that are really just picking one option from a short list, like matching a client request to the right GitHub issue. so i built a weekly client follow-up agent that does exactly that. i tested it on 40 requests, and it was as accurate as letting Opus make every match, at 150x lower decision cost. here's the tech stack i used: โ TrueForge as the agent harness โ Jev for the matching decisions โ Daytona as the sandbox for code execution Jev is a decision model from TypeSafe AI. it takes the options from your app and returns a structured choice with a calibrated confidence score, without generating any text. that confidence decides what gets a second look. low-confidence matches go back to Opus for review, and in my test, that review caught 5 wrong matches that would otherwise have gone through. TrueForge is TrueFoundry's harness, and it keeps context small. it loads tool definitions only when needed, writes large tool outputs to files, and lets the agent process data in scripts instead of the conversation. on DevRev's Enterprise Bench, it solved about as many tasks as Claude Managed Agents on roughly a third of the tokens. it also keeps credentials outside the sandbox and holds every write for approval. if you want to try this out, TrueForge is fully open source (MIT-licensed) and runs on your own machine with a single command. TrueForge GitHub: (don't forget to star ๐) to help you reproduce my work and understand all the integration details, i wrote a detailed article. it's quoted below.
Akshay ๐16,530 Aufrufe โข vor 4 Tagen
Another blow to Anthropic! They spent months building what's now fully open-source. Anthropic recently put Claude inside Slack, where you can tag it in a channel. It reads the thread, breaks the task into steps, and posts the result back. The problem is that it only runs Claude and only in the channels Anthropic supports. Running your own agent there is harder. The reasoning, tool calls, and state management are mostly handled by the framework. Connecting that agent to a messaging platform is not. Moreover, each platform has a different integration: - Slack renders messages with Block Kit - Teams uses Adaptive Cards - and each has its own SDK, auth flow, and delivery model. If an agent needs to run on three platforms, one must write three separate integrations against the same agent logic. That overhead explains why most custom agents never get deployed to Slack, and why the ones that do are usually a single vendor's hosted assistant. The alternative is to keep the agent in one place and add a per-platform adapter that translates its output into each platform's native format. The agent is written once, and each channel requires just another output target instead of a separate build. CopilotKit open-sourced this full implementation in the Channels SDK. Essentially, any agent that implements AG-UI can run in a messaging platform in a few lines of code, like Slack, Teams, Discord, WhatsApp, and many more. Because the agent runs inside the thread, it has that conversation's context, so it can summarize the discussion, open a ticket, or route to the right person. It works with any backend, so LangGraph, CrewAI, Mastra, Google ADK, or a plain HTTP agent can connect through an existing endpoint. The same message can render as a Block Kit in Slack and as Adaptive Cards in Teams. In practice, the model and orchestration stay the same; it requires no migration or rewrite. It also handles human-in-the-loop approvals, persistence, and transcripts that carry state across platforms, so a thread started in Teams can continue in Slack. CopilotKit is open-source, and AG-UI is supported across every major agent framework, including LangGraph, CrewAI, Mastra, and Google ADK. Here's the repo: (don't forget to star it โญ) The agent running in Slack no longer has to be a vendor's. It can be the one you already built. The video below shows this in action. Thanks to CopilotKit for working with me on this launch.
Akshay ๐244,645 Aufrufe โข vor 2 Monaten
This is the DeepSeek moment for Voice AI. Chatterbox Turbo is an MIT-licensed voice model that beats ElevenLabs Turbo & Cartesia Sonic 3! - <150ms time-to-first-sound - Voice cloning from just 5-second audio - Paralinguistic tags for real human expression 100% open-source.
Akshay ๐469,213 Aufrufe โข vor 9 Monaten
How to build a 1-person AI company that: - Runs locally - 100% open-source - No human employees, all agents - Real-time collaboration via email Multi-agent orchestration is not new. Plenty of frameworks already let agents hand off tasks, run in parallel, and talk to each other. So the interesting question is not whether agents can collaborate. It is what structure you use to make them collaborate. The common approach is to wire a graph of nodes and edges and reason about the plumbing yourself. It works, but you are learning a new abstraction just to describe who does what. There is a coordination structure we have trusted for a hundred years already: an organization. Every company runs the same way. People have roles, roles have reporting lines, and work moves up and down that chart without anyone relaying each message by hand. Map that onto agents and the whole thing gets intuitive. You lay out an org chart, each agent fills one role, you talk to the person at the top, and the org sorts out the work between them. You already know how a company works, so you already know how to run one here. There is no new abstraction to learn. That is exactly what Alook does. Each agent is a live Claude Code or OpenCode session with a defined role, a reporting line, and its own email inbox. The agents coordinate over email, the same way a team would. And it all runs locally through a runtime on your own machine, so nothing leaves your setup. You bring your own agent too. Claude Code and Codex both work, and if you would rather stay fully open source and local, OpenCode works the same way. To show how this feels in practice, I set up three agents as a small sales team. Vi is the one I talk to. I hand Vi a goal, and Vi routes the work down the chart. Neile runs prospect research. Vi passes the target criteria, and Neile reports back a ranked list of names, roles, and companies, each with a suggested angle and a confidence score. Lliane runs outreach. Vi hands over the messaging angle and follow-up cadence, and Lliane reports back on emails sent, responses received, and any deal that needs escalation. I never relay a message between them. Neile and Lliane report to Vi, and Vi updates me in one place. The whole thing is open source and self-hosted, so it runs on your machine with your own agents. Give the repo a star if you want to follow where it goes: I also wrote a full walkthrough on building your own AI company with it, from a blank org chart to a running job. The article is quoted below. Cheers! :)
Akshay ๐170,853 Aufrufe โข vor 3 Monaten
LLM Routing, clearly explained! (routing can cost more than not routing at all) The basic idea sounds simple: Send easy requests to cheaper models and reserve frontier models for difficult tasks. But production routing is rarely that straightforward. Every classification adds latency. General-purpose models can misread intent. Routing rules become outdated. And switching models during an agent session can destroy prefix-cache savings, increase costs, and create inconsistent behavior. In this video, I cover: โ Why DIY routing often breaks in production โ How prefix caching changes the economics โ Why agents need session pinning and model affinity โ How Plano and Arch-Router handle intent resolution โ How to build, test, and evaluate a router with DigitalOcean Routing works, but only when the infrastructure accounts for the full session, not just the latest request. The real question is not whether routing can reduce costs. It is how much of your AI traffic has been overpaying this whole time. Get started with Digital Ocean inference router: I recently wrote a detailed article on the same topic. The article is quoted below.
Akshay ๐16,223 Aufrufe โข vor 10 Tagen
Sam Altman made the case for open-source harnesses in July. a month later, someone shipped it, and it's more efficient than most managed harnesses. here is the problem it was aimed at: a large share of your agent's token bill is the model rereading things it already read. that isn't the model's doing. the runtime around it decides what goes into every prompt and how often the model gets called. for example, an agent queries a CRM at step four and gets back 400 rows. those rows get piled up in the conversation history. by step nineteen, the model has to read those rows fifteen times unnecessarily, and every token read is billed at input rates. it happened because your harness assembled that prompt on every turn and kept the rows in it. that gives you two levers: how much context the harness carries forward, and how often it calls the model. there are four practical ways to keep the prompt from growing unnecessarily: โ load tool schemas on demand. a server with 100 tools doesn't need to put all 100 into every prompt when the agent only calls two. โ offload large results to disk. turn a large response into a short preview and a file path instead of replaying the entire result on every turn. โ delegate to subagents. let a subagent spend thirty tool calls in its own context and return one summary to the root agent. โ run toolchains in code. one script calls three tools, joins the results, and returns a table instead of three turns each dragging a full response. but reducing context is only half the job. you also need to control how often the model gets called. a good harness should avoid unnecessary planning, verification, and reflection when the work can be completed in fewer steps. TrueFoundry's open-source agent harness, TrueForge, is built around both of those controls. it sits between the model and the tools, deciding what goes into every prompt and when another model call is actually needed. it also breaks token usage down across the harness, skills, instructions, tools, and messages. DevRev's Enterprise-Bench is where this gets tested, on multi-step tasks of the kind where an agent pulls records from one system and reconciles them against another. TrueFoundry ran TrueForge there against Claude Managed Agents, both on the same model, and both finished the same number of tasks. the tie is the part that matters, because it means the gap underneath is not a quality tradeoff. TrueForge reached that score on close to a third of the tokens, with roughly 40% fewer trips back to the model. for the same result, that comes out around 2.7x cheaper than Claude Managed Agents. swapping in an open model made it sharper still. TrueForge with GLM-5.2 scored a little higher than either setup above, and the entire benchmark run cost about $3 at list prices. being open source matters beyond the license here. the model underneath can be swapped without rewriting the agent, and the whole thing can run inside your own environment when the data cannot leave it. all of this comes down to the runtime around the model, the context it carries, the tools it exposes, and how many times it goes back to the model. that is what a production harness actually owns. the full task list, the per-run numbers, and the MIT-licensed code are on GitHub: (don't forget to star ๐) you can read more about the same in the article quoted below. thanks to the TrueForge team for working with me on this one.
Akshay ๐76,998 Aufrufe โข vor 1 Monat
run agent harnesses 100% private & offline. (no token costs, no API keys, 100% open-source) your agent runs locally. the model doesn't. every prompt, every file, and every secret still leaves your machine before the agent does anything with it. Magnitude fixes that. it's an open source inference server that runs models on your own hardware and plugs into the coding agent you already use. setup is one command. it profiles your machine, measures the memory bandwidth that sets your token rate, and hands back complete configurations instead of a list of models. each one names a model, a compression level, a context size, and a speed range you can expect. pick one and start working. it doesn't replace your harness. setup asks which one you want and writes that config for you. Pi, OpenCode, Claude Code, Codex, and Cline all work, and there's a built-in one tuned for local models if you don't have a harness yet. that one uses your shell, edits files, and runs scripts out of the box. add skills and it handles Excel, PowerPoint, PDFs, or Chrome. everyday work it covers: โ analyze sensitive data โ manage private notes โ review code and logs โ search and organize files โ build docs or slides Apache 2.0. no rate limits, and nothing leaves the machine. ๐ป๐ฝ๐บ ๐ถ -๐ด @๐บ๐ฎ๐ด๐ป๐ถ๐๐๐ฑ๐ฒ๐ฑ๐ฒ๐/๐ฐ๐น๐ถ the repo is here: (don't forget to star ๐) i wrote the full breakdown of why picking the configuration is the hard part. the article is quoted below.
Akshay ๐56,499 Aufrufe โข vor 1 Monat
Anthropic won't like this open-source repo. It is going to cost LLM providers a lot of money. Every CI run of an AI app today sends real requests to providers like OpenAI or Anthropic. Like any other LLM call, this too gets billed at actual API rates. So for teams with high commit volumes, this accumulates into a meaningful chunk of API spend. One common hack devs use is that instead of invoking the LLM API, the test calls a fake local server that speaks the same API and returns a dummy response. The catch is that the dummy response is a copy of what the provider returned on the day it was saved, and providers keep adding fields and changing types. So the tests keep passing against a schema that's no longer valid, while the real integration breaks in production. A smart approach is now actually implemented in CopilotKit๐ช's recently open-sourced aimock project. Every day, the repo's own CI sends a handful of requests to the real API and the same requests to the fake server, then compares both against the official client library's type definitions. Those are the only real API calls in the whole setup, and they run on the repo's own keys, not in anyone else's CI. A single team can push hundreds of commits a day, and thousands of teams are already doing that with coding agents. All of those runs stay offline, because one repo checks against the real API on everyone's behalf. When a check fails, a coding agent updates aimock's built-in response schema, the full test suite has to pass, and a patch version ships to npm. By simply upgrading the package, the corrected schema gets reflected in every project using it. The capability is not just limited to a single provider. The same server works for Claude, OpenAI, Gemini, Bedrock, Azure, Ollama, plus MCP tools, A2A agents, AG-UI event streams, vector DBs like Pinecone and Qdrant, and search, speech, image, and video endpoints. Here's the repo: (don't forget to star it โญ) That said, mocking your API calls is one thing. AI engineers should also know how to test agents properly in the first place, which several teams still skip. I wrote a full walkthrough on that, covering build, testing, evals, tracing, and deployment. Read it below.
Akshay ๐62,821 Aufrufe โข vor 2 Monaten