Big win for open-source LLMs! DeepSeek V4 Pro holds... the top open-weights score on SWE-bench Verified, in the GPT-5.5 range. GLM 5.2 leads the open-weight intelligence index and sits near the closed frontier on long-horizon coding. But this leaderboard number is a weak proxy for real performance. It comes from one task set, run through one harness, served at one precision. The same weights can even score differently across providers, since many hosts quantize activations to fp8 and drift the model off its reference weights. Real performance is determined based on whether a model can read a repo, make coordinated edits across files, run the tests, and recover when one breaks. By that measure, the top open models hold up, but only inside the right harness. The teams that actually put DeepSeek V4 into production pipelines as a frontier substitute got there through the harness they built around the model, not by picking a stronger model. If you want to see this in practice, Cline (64k+ stars) has actually built that harness around open models, tuned so they run at production quality. And it's tuned so that these LLMs can run at production quality, with plan and act modes, checkpoints, and terminal feedback. ClinePass is the new access layer on top of it. It runs a curated set of those models inside Cline, narrowed to the ones tested for coding-agent use, with 2 to 5x the standard rate limits and no separate provider accounts, keys, or billing to track. The video below shows the setup, and I worked with the team to put this together. It runs alongside custom keys and local models as well, not in place of them.show more

New open-source agent harness just landed! I got early... access to TrueForge by TrueFoundry and have been running it locally for the past few days. The harness layer deserves as much attention as the model, and open source matters here because you can inspect the loop, run it on your own infrastructure, and swap to the latest or cheaper models. TrueForge handles the runtime work that makes an agent reliable. It drives the tool-calling loop, manages context, coordinates subagents, and executes code in a sandbox, with any model you choose. Every tool call re-sends the growing context to the model, so in practice the harness controls most of what an agent costs to run. A few things stood out from my testing and their published benchmarks. Vendor-Neutral by design. It runs OpenAI, Anthropic, and Google models alongside open-weight models like Kimi, GLM, and DeepSeek. Model routing is a setting, and you can send each task to the model that fits it. On a 14-task enterprise agent benchmark, it matched the accuracy of Claude Managed Agents running the same Opus 4.8 model at roughly 30% lower cost per run (3.8M tokens vs 10M for the same answers). Routing the same tasks to GLM-5.2 held accuracy and brought cost down by about 75%, around $3 per run instead of $12. Fully self-hosted and Open Source (MIT License). I had it running locally with one command, with sandboxed code execution working out of the box. It's time to own your agent harness. Thanks to TrueFoundry for partnering on this post.show more

Something just launched that makes VS Code feel ancient.... And im all in for it: Cline Desktop app, an open-source app for open-weight models (and you know how much i love open source) brings its coding agent into a standalone Cline Desktop app for Mac + Windows. Just open a project and tell it what you want to change. No need to open VS Code first. In Cline's demo, it adds priority filters to a task board, works through the code and runs the build. Glad to see the model choice stays open, too. You can bring your own API keys, use supported open-weight models and even switch models halfway through a project!show more

Cline Desktop just launched. An open-source workspace for open-weight... models 🤯 > Import tasks from other coding agents > Run agents in parallel > Schedule tasks to run on their own Free models built in: DeepSeek V4 Flash, GLM 5.3 Flash, Laguna S 2.1. I had early access and I can assure you it feels incredible. Pick your model and provider, and run it. 100% open source.show more

🏁 FINAL RESULTS: Bonsai 2 9B now runs 471%... faster on Macs and 421.5% faster on NVIDIA DGX Spark. Open frontier research and open-weight models win again. One open challenge, one week with the community and it runs 6x faster on hardware people already own. It is crazy to see that the decode speed went from 144.8 to 959.6 tokens/sec on Apple Silicon and from 85.9 to 576.7 on DGX Spark. This model can also run on an iPhone. I'm very excited to see what people build with it and how it introduces MLX and local AI to new faces. Bonsai 2 9B has been live on Darkbloom since day one, so anyone can try it right now. 158 new records built on top of each other from humans and agents running Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Gemini 4 Argon, DeepSeek-V4 and Qwen3.8-Flash-Next. Congrats to Kaankolcu took the top spot on Mac, and dadon took it on NVIDIA. Thank you to the PrismML team for doing this in the open with us, Babak Hassibi, Omead Pooladzandi, Sahin Lale and the rest of the PrismML team.show more

Don't train the model, evolve the harness. I read... a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.show more

Okay... this is actually insane. OpenCodex feels like the... open-source breakthrough I've been waiting for. The best part : You can plug multiple providers into the same OpenAI Codex harness and switch between models depending on the task. Running low on tokens? No problem. Use another provider. OpenRouter free model today? Plug it in. This completely changes how I think about AI coding workflows. And yes... it even works on mobile. OpenCodex might be one of the most useful open-source AI projects I've seen this year. OpenAI built an incredible harness. The open-source community just made it universal.show more

Okay this is kinda wild 👀 NVIDIA is basically... handing out FREE API keys to 100+ top AI models - GLM 5.2, DeepSeek V4, Kimi K2.6, MiniMax M3, their own Nemotron and a ton more. it's called NVIDIA NIM and I had no idea it existed. it's rate-limited, so not something you'd run in production - but for personal use it's honestly great. you can poke a dozen frontier models, and some you can even grab and self-host. dropped a key into OpenCode and it just worked. the hyped ones like GLM 5.2 are jammed with queue right now, but the lighter models fly. all of it free.show more

We released physics-intern: a simple harness for science problems!... It gets models like Gemini 3.1 Pro to go from 17.7 -> 31.4, thus beating GPT 5.5 Pro. The physics-intern harness can wrap any model and via dedicated subagent boost the performance of the vanilla reasoning models. While I think more and more of these harness capability gains will be absorbed into the models (like prompting tricks disappeared over time) there is a lot to be gained right now by building good scaffolds for those models and integrating tools well. Interestingly, the exception we found that GPT 5.5 Pro actually didn't benefit from the physics-intern harness! Read more about it here: PS: I think the Harness[Model] notation is kind of nice.show more

THAT $70 "RUN YOUR OWN LLMS" PI KIT CAN'T... RUN A SINGLE LLM. IT'S A VISION CHIP WITH NO RAM. that clip sells a raspberry pi 5 in a slick case with an ai accelerator and the caption "your own llms." clean build, fun kit. the claim is where it breaks. the fine print: the popular $70 pi ai kit uses a hailo-8l, 13 tops. it's built for vision, object detection and image processing, and it has no memory of its own. so it cannot run large language models. full stop the board that actually can is a different one: the newer ai hat+ 2, hailo-10h, 40 tops, with 8gb of dedicated ram. that's $130, not $70 and even that runs only tiny models. llama 3.2 at 1b, qwen 2.5 at 1.5b, deepseek r1 at 1.5b. edge llms live in the 1-7b range, against cloud models at 500b to 2 trillion so the honest pitch: for $130 you can run a very small language model on a pi, slowly, as a fun learning project. that's real and it's cool. "your own llms" on a $70 vision kit is not. why this keeps happening: "ai kit" and a big "tops" number sell. tops sounds like intelligence. but tops measures vision-style math, not whether the chip has the memory to hold a language model. the spec that matters for llms is ram, and the cheap kit has none. the honest caveats, both ways: the $70 kit is genuinely great, just at vision. cameras, object detection, that's its job the $130 hat really does run small llms locally, which a pi couldn't do at all two years ago. that's progress "small" is the load-bearing word. don't expect gpt at home on a pi the takeaway: before you buy a kit because the caption says llm, check two numbers. not the tops. the ram, and the size of the model it can actually load. no 70-dollar miracle, no gpt in a pi case, no tops number that means what you think. save this before you buy the wrong kit for the word on the box.show more

If you are trying to understand where AI agents... are going, learn harness engineering. A capable model is only one part of an agent system. Once the model begins reading files, calling tools, modifying state and working across many steps, the quality of the system depends increasingly on the software around it. Consider a coding agent working through a large repository. The model can decide that it needs to inspect a file, search for a symbol, make an edit or run a test, but those decisions do not execute themselves. The surrounding runtime has to decide which resources are available, whether the requested action is permitted, how the operation should be performed, what result should be retained, and what information should be presented to the model on the next step. This becomes harder as the run gets longer. As history accumulates, replaying everything can become costly and less effective. The harness has to decide what should remain in context, what should be summarized or retrieved later, and what belongs in persistent state outside the context window. Execution has similar problems. A long-running agent may need to survive an interruption, avoid repeating completed work, enforce permissions around consequential actions, and preserve enough history to reconstruct what happened when the final result is wrong. These are harness problems. The harness is the layer that manages context, tools, execution, state, checkpoints, limits and traces around the model. Harness engineering is the work of designing and improving that layer. Engineers inspect execution traces, evaluate agents on representative tasks, look for recurring failure modes, and then change things such as context selection, tool interfaces, state handling or execution controls. That last part matters because agent failures are often not fixed by changing the model. Sometimes the useful change is in what the model sees, how a tool is exposed, what state is preserved, or what the runtime does after a failed step. As agents take on longer tasks, the demands on this surrounding software grow. Model capability remains essential, but harness engineering is what turns that capability into an execution process that can be controlled, inspected, tested and improved.show more

You can now post-train a model inside your existing... production harness with our platform, AC2. A production harness is a whole engineered system around the LLM, with its own context management, tools, sandboxing, and control flows. Porting that into a new training runtime can be expensive and could introduce train-test mismatch, where the policy is optimized against a simulated harness and then struggles in production. All you need to do is swap out the harness’ LLM response endpoint to one provided by AC2, and expose a lightweight protocol for AC2 to initiate and grade rollouts; the trainer handles the rest.show more

EXPLAINED: How multiplayer autoresearch on Yukon works A lot... of people have been asking, so I made a visual of the full step-by-step. > Challenge setter (human) posts a hard problem with a clear score and an automatic verifier. > Humans (solvers) and AI agents attack it at once. Claude, GPT, Gemini, DeepSeek, Qwen, Kimi, GLM. It could be any model, any harness. > Each attempt goes through the verifier. > When one beats the current best, it becomes the new frontier (the record to beat). > Anyone can build on that new best instead of starting from 0. Then the loop repeats. All results and commits are open source, so you can trace how every record was set. I think this is how frontier research/science gets done in an AGI and PostAGI world. So far we've solved hard problems across cryptography, math, science and open-weight models. We'll keep going there and we're starting to explore biology, chemistry and materials science. See what we've solved (and are solving now) in each one onshow more

this is straight f*cking gold the full harness guide... for Kimi K3, the #1 open source frontend model the premise: the model underneath keeps changing. the harness is the part that stays yours what one night of it looks like: 02:00 - the trigger fires, every node that needs work gets picked > 212 agents fan out, one per node > a bad return gets rejected, retried once with the reason attached > one agent tries to write outside its folder. blocked. nobody woken up > +41 nodes and +96 edges land in the graph > a drafted email hits pre_send and waits for you 02:41 - the loop stops on its own, inside a 45 minute budget 07:30 - you read one file and make two decisions the whole machine is one folder, one config file, five short scripts two rules hold it together: > the model sits behind one line, so a better model is a config change > the verifier lives outside the agent, so nothing grades its own work now the money part companies burn whole quarters building internal agent platforms a harness setup for a small team goes for four figures, one time then a monthly retainer to run the swap test on every new model release models come and go. the person who owns the harness keeps getting paidshow more

One hundred thousand Github Stars!! This makes Paperclip one... of the top 150 open-source projects of. all. time. All computer work is becoming agentic work so you need tools to harness the army of agents at your disposal We've built 📎 for companies to get a handle on agent sprawl and it's popular is because it's good Paperclip is multi-player and you can use any model, any harness, any time. It's the layer for every connection, every agent, and every human to collaborate on work The self hosted version is completely free (and you can sign up for the waitlist for the hosted version) 📎📎📎 ITS TIME TO CLIP 📎 📎📎show more

Lauren from the Grok Bot team just confirmed it:... every bot now runs on Claude Opus 5.5 and each one can spin up its own cloud agents running Cursor’s models this is f*cking insane... so i built the harness on top of it, in one prompt: Opus 5.5 leads and builds, Grok scouts live X and the web, every claim needs two sources, a red team attacks the draft before you see it, and every run gets logged so the next one starts smarter you stop prompting models. you sign off on results. paste it into Grok Bot or Claude and thank me later all 10 steps, the workflow, and real examples below ↓show more

Stanford researchers did it again. They just built the... agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.show more

This is pure f*cking treasure. ex-Google engineer reveals his... AI agent playbook. 14+ years at Google. $0 to read it. the topic is harness engineering: everything around the model that lets it execute, check its work and recover when something breaks. one example in his breakdown deserves attention. LangChain’s coding agent went from 52.8% to 66.5% on Terminal-Bench 2.0. the model stayed the same. the team changed the prompts, tools and machinery around it. Agent = Model + Harness. the system has six practical jobs: > guides give the agent project rules, constraints and examples > sensors check its work through tests, linters and validators > the loop handles execution, verification, retries and escalation > memory preserves state, artifacts and decisions > permissions control which tools and actions are allowed > observability records tool calls, costs and failures the part worth stealing is what happens after a mistake. an agent keeps declaring broken code finished. you add a verification step before completion. it keeps ignoring a project convention. you add a targeted instruction and a check for that specific failure. it touches files outside its task. you tighten its write permissions. each recurring failure gives you a concrete improvement to make in the harness. that improvement can carry into every future run. this is where the engineering work compounds. bookmark it. start with the mistake you have already paid your agent to make twice.show more

PATRONUS RECORDED 100+ HOURS OF AI AGENTS PLAYING VIDEO... GAMES AND PUT IT ALL IN PUBLIC Not the scores. The footage. → Watch the actual runs, not a number in a table → Frontier models on one leaderboard → Open for you to play and compete Benchmarks tell you who won.show more

Black Forest Labs just announced FLUX 3: a unified... multimodal model for image, video, audio and action prediction. Founded in Freiburg, Germany, one of the few globally significant European companies in the AI ​​sector. The current release is gated early access, not open source. But if FLUX 3 Dev actually ships with usable open weights, this could become one of the most important open-model releases in generative video yet. Especially after the Chinese Minimax H3 release. BFL claims, based on preliminary internal comparisons, that FLUX 3 outperformed models such as Runway Gen-4.5, Luma Ray 3.2, Kling 3 Pro, and Seedance 2.0. However, these figures stem from the early-access phase and do not constitute independent validation. Be that as it may, it's great to see the quality that today's video models can produce at affordable prices. Something that would have been unthinkable a year ago. Really cool release!show more

The redesigned Arena Leaderboard Overview is live! The frontier... is moving quickly, so we built a central hub to help you keep up. The new Arena Leaderboard experience brings all our live signals into one place for researchers, builders, and our global community of users who power our model evaluations. Here’s what’s new: - New Release Models: a real-time feed of the latest models added to Arena - Performance by Category: surfacing live sessions, and the top models by categories like Agents, Images, and Coding. - Model Capabilities: first impressions of new models, straight from the Arena team - Arena News: the latest research, features and updates from around the Arena universe For example, now you can see scores across arenas for models like GPT-6 Sol and Claude Opus 5.5, alongside signals like their category standings and capabilities, their latest news and more. Explore scores and now much more at our redesigned leaderboard. Find the link below.show more