正在加载视频...

视频加载失败

返回首页

Big win for open-source LLMs! DeepSeek V4 Pro holds the top open-weights score on SWE-bench Verified, in the GPT-5.5 range. GLM 5.2 leads the open-weight intelligence index and sits near the closed frontier on long-horizon coding. But this leaderboard number is a weak proxy for real performance. It comes...

_avichawla's profile picture

Avi Chawla

72,646 subscribers

44,124 次观看 • 4 个月前 •via X (Twitter)

科学技术教育
LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

New open-source agent harness just landed! I got early access to TrueForge by TrueFoundry and have been running it locally for the past few days. The harness layer deserves as much attention as the model, and open source matters here because you can inspect the loop, run it on your own infrastructure, and swap to the latest or cheaper models. TrueForge handles the runtime work that makes an agent reliable. It drives the tool-calling loop, manages context, coordinates subagents, and executes code in a sandbox, with any model you choose. Every tool call re-sends the growing context to the model, so in practice the harness controls most of what an agent costs to run. A few things stood out from my testing and their published benchmarks. Vendor-Neutral by design. It runs OpenAI, Anthropic, and Google models alongside open-weight models like Kimi, GLM, and DeepSeek. Model routing is a setting, and you can send each task to the model that fits it. On a 14-task enterprise agent benchmark, it matched the accuracy of Claude Managed Agents running the same Opus 4.8 model at roughly 30% lower cost per run (3.8M tokens vs 10M for the same answers). Routing the same tasks to GLM-5.2 held accuracy and brought cost down by about 75%, around $3 per run instead of $12. Fully self-hosted and Open Source (MIT License). I had it running locally with one command, with sandboxed code execution working out of the box. It's time to own your agent harness. Thanks to TrueFoundry for partnering on this post.
0:16

New open-source agent harness just landed! I got early access to TrueForge by TrueFoundry and have been running it locally for the past few days. The harness layer deserves as much attention as the model, and open source matters here because you can inspect the loop, run it on your own infrastructure, and swap to the latest or cheaper models. TrueForge handles the runtime work that makes an agent reliable. It drives the tool-calling loop, manages context, coordinates subagents, and executes code in a sandbox, with any model you choose. Every tool call re-sends the growing context to the model, so in practice the harness controls most of what an agent costs to run. A few things stood out from my testing and their published benchmarks. Vendor-Neutral by design. It runs OpenAI, Anthropic, and Google models alongside open-weight models like Kimi, GLM, and DeepSeek. Model routing is a setting, and you can send each task to the model that fits it. On a 14-task enterprise agent benchmark, it matched the accuracy of Claude Managed Agents running the same Opus 4.8 model at roughly 30% lower cost per run (3.8M tokens vs 10M for the same answers). Routing the same tasks to GLM-5.2 held accuracy and brought cost down by about 75%, around $3 per run instead of $12. Fully self-hosted and Open Source (MIT License). I had it running locally with one command, with sandboxed code execution working out of the box. It's time to own your agent harness. Thanks to TrueFoundry for partnering on this post.

elvis

11,303 次观看 • 2 个月前

Something just launched that makes VS Code feel ancient. And im all in for it: Cline Desktop app, an open-source app for open-weight models (and you know how much i love open source) brings its coding agent into a standalone Cline Desktop app for Mac + Windows. Just open a project and tell it what you want to change. No need to open VS Code first. In Cline's demo, it adds priority filters to a task board, works through the code and runs the build. Glad to see the model choice stays open, too. You can bring your own API keys, use supported open-weight models and even switch models halfway through a project!
0:17

Something just launched that makes VS Code feel ancient. And im all in for it: Cline Desktop app, an open-source app for open-weight models (and you know how much i love open source) brings its coding agent into a standalone Cline Desktop app for Mac + Windows. Just open a project and tell it what you want to change. No need to open VS Code first. In Cline's demo, it adds priority filters to a task board, works through the code and runs the build. Glad to see the model choice stays open, too. You can bring your own API keys, use supported open-weight models and even switch models halfway through a project!

Chubby♨️

40,292 次观看 • 1 个月前

Cline Desktop just launched. An open-source workspace for open-weight models 🤯 > Import tasks from other coding agents > Run agents in parallel > Schedule tasks to run on their own Free models built in: DeepSeek V4 Flash, GLM 5.3 Flash, Laguna S 2.1. I had early access and I can assure you it feels incredible. Pick your model and provider, and run it. 100% open source.
0:10

Cline Desktop just launched. An open-source workspace for open-weight models 🤯 > Import tasks from other coding agents > Run agents in parallel > Schedule tasks to run on their own Free models built in: DeepSeek V4 Flash, GLM 5.3 Flash, Laguna S 2.1. I had early access and I can assure you it feels incredible. Pick your model and provider, and run it. 100% open source.

Alvaro Cintas

15,934 次观看 • 1 个月前

🏁 FINAL RESULTS: Bonsai 2 9B now runs 471% faster on Macs and 421.5% faster on NVIDIA DGX Spark. Open frontier research and open-weight models win again. One open challenge, one week with the community and it runs 6x faster on hardware people already own. It is crazy to see that the decode speed went from 144.8 to 959.6 tokens/sec on Apple Silicon and from 85.9 to 576.7 on DGX Spark. This model can also run on an iPhone. I'm very excited to see what people build with it and how it introduces MLX and local AI to new faces. Bonsai 2 9B has been live on Darkbloom since day one, so anyone can try it right now. 158 new records built on top of each other from humans and agents running Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Gemini 4 Argon, DeepSeek-V4 and Qwen3.8-Flash-Next. Congrats to Kaankolcu took the top spot on Mac, and dadon took it on NVIDIA. Thank you to the PrismML team for doing this in the open with us, Babak Hassibi, Omead Pooladzandi, Sahin Lale and the rest of the PrismML team.
0:15

🏁 FINAL RESULTS: Bonsai 2 9B now runs 471% faster on Macs and 421.5% faster on NVIDIA DGX Spark. Open frontier research and open-weight models win again. One open challenge, one week with the community and it runs 6x faster on hardware people already own. It is crazy to see that the decode speed went from 144.8 to 959.6 tokens/sec on Apple Silicon and from 85.9 to 576.7 on DGX Spark. This model can also run on an iPhone. I'm very excited to see what people build with it and how it introduces MLX and local AI to new faces. Bonsai 2 9B has been live on Darkbloom since day one, so anyone can try it right now. 158 new records built on top of each other from humans and agents running Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Gemini 4 Argon, DeepSeek-V4 and Qwen3.8-Flash-Next. Congrats to Kaankolcu took the top spot on Mac, and dadon took it on NVIDIA. Thank you to the PrismML team for doing this in the open with us, Babak Hassibi, Omead Pooladzandi, Sahin Lale and the rest of the PrismML team.

Kydo

13,658 次观看 • 10 天前

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.
0:18

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.

Akshay 🚀

245,707 次观看 • 4 个月前

Okay... this is actually insane. OpenCodex feels like the open-source breakthrough I've been waiting for. The best part : You can plug multiple providers into the same OpenAI Codex harness and switch between models depending on the task. Running low on tokens? No problem. Use another provider. OpenRouter free model today? Plug it in. This completely changes how I think about AI coding workflows. And yes... it even works on mobile. OpenCodex might be one of the most useful open-source AI projects I've seen this year. OpenAI built an incredible harness. The open-source community just made it universal.
0:10

Okay... this is actually insane. OpenCodex feels like the open-source breakthrough I've been waiting for. The best part : You can plug multiple providers into the same OpenAI Codex harness and switch between models depending on the task. Running low on tokens? No problem. Use another provider. OpenRouter free model today? Plug it in. This completely changes how I think about AI coding workflows. And yes... it even works on mobile. OpenCodex might be one of the most useful open-source AI projects I've seen this year. OpenAI built an incredible harness. The open-source community just made it universal.

CHOI

42,894 次观看 • 3 个月前

Okay this is kinda wild 👀 NVIDIA is basically handing out FREE API keys to 100+ top AI models - GLM 5.2, DeepSeek V4, Kimi K2.6, MiniMax M3, their own Nemotron and a ton more. it's called NVIDIA NIM and I had no idea it existed. it's rate-limited, so not something you'd run in production - but for personal use it's honestly great. you can poke a dozen frontier models, and some you can even grab and self-host. dropped a key into OpenCode and it just worked. the hyped ones like GLM 5.2 are jammed with queue right now, but the lighter models fly. all of it free.
0:15

Okay this is kinda wild 👀 NVIDIA is basically handing out FREE API keys to 100+ top AI models - GLM 5.2, DeepSeek V4, Kimi K2.6, MiniMax M3, their own Nemotron and a ton more. it's called NVIDIA NIM and I had no idea it existed. it's rate-limited, so not something you'd run in production - but for personal use it's honestly great. you can poke a dozen frontier models, and some you can even grab and self-host. dropped a key into OpenCode and it just worked. the hyped ones like GLM 5.2 are jammed with queue right now, but the lighter models fly. all of it free.

Stefan 3D AI

26,272 次观看 • 3 个月前

We released physics-intern: a simple harness for science problems! It gets models like Gemini 3.1 Pro to go from 17.7 -> 31.4, thus beating GPT 5.5 Pro. The physics-intern harness can wrap any model and via dedicated subagent boost the performance of the vanilla reasoning models. While I think more and more of these harness capability gains will be absorbed into the models (like prompting tricks disappeared over time) there is a lot to be gained right now by building good scaffolds for those models and integrating tools well. Interestingly, the exception we found that GPT 5.5 Pro actually didn't benefit from the physics-intern harness! Read more about it here: PS: I think the Harness[Model] notation is kind of nice.
0:12

We released physics-intern: a simple harness for science problems! It gets models like Gemini 3.1 Pro to go from 17.7 -> 31.4, thus beating GPT 5.5 Pro. The physics-intern harness can wrap any model and via dedicated subagent boost the performance of the vanilla reasoning models. While I think more and more of these harness capability gains will be absorbed into the models (like prompting tricks disappeared over time) there is a lot to be gained right now by building good scaffolds for those models and integrating tools well. Interestingly, the exception we found that GPT 5.5 Pro actually didn't benefit from the physics-intern harness! Read more about it here: PS: I think the Harness[Model] notation is kind of nice.

Leandro von Werra

97,519 次观看 • 5 个月前

THAT $70 "RUN YOUR OWN LLMS" PI KIT CAN'T RUN A SINGLE LLM. IT'S A VISION CHIP WITH NO RAM. that clip sells a raspberry pi 5 in a slick case with an ai accelerator and the caption "your own llms." clean build, fun kit. the claim is where it breaks. the fine print: the popular $70 pi ai kit uses a hailo-8l, 13 tops. it's built for vision, object detection and image processing, and it has no memory of its own. so it cannot run large language models. full stop the board that actually can is a different one: the newer ai hat+ 2, hailo-10h, 40 tops, with 8gb of dedicated ram. that's $130, not $70 and even that runs only tiny models. llama 3.2 at 1b, qwen 2.5 at 1.5b, deepseek r1 at 1.5b. edge llms live in the 1-7b range, against cloud models at 500b to 2 trillion so the honest pitch: for $130 you can run a very small language model on a pi, slowly, as a fun learning project. that's real and it's cool. "your own llms" on a $70 vision kit is not. why this keeps happening: "ai kit" and a big "tops" number sell. tops sounds like intelligence. but tops measures vision-style math, not whether the chip has the memory to hold a language model. the spec that matters for llms is ram, and the cheap kit has none. the honest caveats, both ways: the $70 kit is genuinely great, just at vision. cameras, object detection, that's its job the $130 hat really does run small llms locally, which a pi couldn't do at all two years ago. that's progress "small" is the load-bearing word. don't expect gpt at home on a pi the takeaway: before you buy a kit because the caption says llm, check two numbers. not the tops. the ram, and the size of the model it can actually load. no 70-dollar miracle, no gpt in a pi case, no tops number that means what you think. save this before you buy the wrong kit for the word on the box.
0:17

THAT $70 "RUN YOUR OWN LLMS" PI KIT CAN'T RUN A SINGLE LLM. IT'S A VISION CHIP WITH NO RAM. that clip sells a raspberry pi 5 in a slick case with an ai accelerator and the caption "your own llms." clean build, fun kit. the claim is where it breaks. the fine print: the popular $70 pi ai kit uses a hailo-8l, 13 tops. it's built for vision, object detection and image processing, and it has no memory of its own. so it cannot run large language models. full stop the board that actually can is a different one: the newer ai hat+ 2, hailo-10h, 40 tops, with 8gb of dedicated ram. that's $130, not $70 and even that runs only tiny models. llama 3.2 at 1b, qwen 2.5 at 1.5b, deepseek r1 at 1.5b. edge llms live in the 1-7b range, against cloud models at 500b to 2 trillion so the honest pitch: for $130 you can run a very small language model on a pi, slowly, as a fun learning project. that's real and it's cool. "your own llms" on a $70 vision kit is not. why this keeps happening: "ai kit" and a big "tops" number sell. tops sounds like intelligence. but tops measures vision-style math, not whether the chip has the memory to hold a language model. the spec that matters for llms is ram, and the cheap kit has none. the honest caveats, both ways: the $70 kit is genuinely great, just at vision. cameras, object detection, that's its job the $130 hat really does run small llms locally, which a pi couldn't do at all two years ago. that's progress "small" is the load-bearing word. don't expect gpt at home on a pi the takeaway: before you buy a kit because the caption says llm, check two numbers. not the tops. the ram, and the size of the model it can actually load. no 70-dollar miracle, no gpt in a pi case, no tops number that means what you think. save this before you buy the wrong kit for the word on the box.

RetroChainer

11,100 次观看 • 3 个月前

If you are trying to understand where AI agents are going, learn harness engineering. A capable model is only one part of an agent system. Once the model begins reading files, calling tools, modifying state and working across many steps, the quality of the system depends increasingly on the software around it. Consider a coding agent working through a large repository. The model can decide that it needs to inspect a file, search for a symbol, make an edit or run a test, but those decisions do not execute themselves. The surrounding runtime has to decide which resources are available, whether the requested action is permitted, how the operation should be performed, what result should be retained, and what information should be presented to the model on the next step. This becomes harder as the run gets longer. As history accumulates, replaying everything can become costly and less effective. The harness has to decide what should remain in context, what should be summarized or retrieved later, and what belongs in persistent state outside the context window. Execution has similar problems. A long-running agent may need to survive an interruption, avoid repeating completed work, enforce permissions around consequential actions, and preserve enough history to reconstruct what happened when the final result is wrong. These are harness problems. The harness is the layer that manages context, tools, execution, state, checkpoints, limits and traces around the model. Harness engineering is the work of designing and improving that layer. Engineers inspect execution traces, evaluate agents on representative tasks, look for recurring failure modes, and then change things such as context selection, tool interfaces, state handling or execution controls. That last part matters because agent failures are often not fixed by changing the model. Sometimes the useful change is in what the model sees, how a tool is exposed, what state is preserved, or what the runtime does after a failed step. As agents take on longer tasks, the demands on this surrounding software grow. Model capability remains essential, but harness engineering is what turns that capability into an execution process that can be controlled, inspected, tested and improved.
0:12

If you are trying to understand where AI agents are going, learn harness engineering. A capable model is only one part of an agent system. Once the model begins reading files, calling tools, modifying state and working across many steps, the quality of the system depends increasingly on the software around it. Consider a coding agent working through a large repository. The model can decide that it needs to inspect a file, search for a symbol, make an edit or run a test, but those decisions do not execute themselves. The surrounding runtime has to decide which resources are available, whether the requested action is permitted, how the operation should be performed, what result should be retained, and what information should be presented to the model on the next step. This becomes harder as the run gets longer. As history accumulates, replaying everything can become costly and less effective. The harness has to decide what should remain in context, what should be summarized or retrieved later, and what belongs in persistent state outside the context window. Execution has similar problems. A long-running agent may need to survive an interruption, avoid repeating completed work, enforce permissions around consequential actions, and preserve enough history to reconstruct what happened when the final result is wrong. These are harness problems. The harness is the layer that manages context, tools, execution, state, checkpoints, limits and traces around the model. Harness engineering is the work of designing and improving that layer. Engineers inspect execution traces, evaluate agents on representative tasks, look for recurring failure modes, and then change things such as context selection, tool interfaces, state handling or execution controls. That last part matters because agent failures are often not fixed by changing the model. Sometimes the useful change is in what the model sees, how a tool is exposed, what state is preserved, or what the runtime does after a failed step. As agents take on longer tasks, the demands on this surrounding software grow. Model capability remains essential, but harness engineering is what turns that capability into an execution process that can be controlled, inspected, tested and improved.

Tech with Mak

49,903 次观看 • 1 个月前

You can now post-train a model inside your existing production harness with our platform, AC2. A production harness is a whole engineered system around the LLM, with its own context management, tools, sandboxing, and control flows. Porting that into a new training runtime can be expensive and could introduce train-test mismatch, where the policy is optimized against a simulated harness and then struggles in production. All you need to do is swap out the harness’ LLM response endpoint to one provided by AC2, and expose a lightweight protocol for AC2 to initiate and grade rollouts; the trainer handles the rest.
0:13

You can now post-train a model inside your existing production harness with our platform, AC2. A production harness is a whole engineered system around the LLM, with its own context management, tools, sandboxing, and control flows. Porting that into a new training runtime can be expensive and could introduce train-test mismatch, where the policy is optimized against a simulated harness and then struggles in production. All you need to do is swap out the harness’ LLM response endpoint to one provided by AC2, and expose a lightweight protocol for AC2 to initiate and grade rollouts; the trainer handles the rest.

Applied Compute

182,521 次观看 • 2 个月前

EXPLAINED: How multiplayer autoresearch on Yukon works A lot of people have been asking, so I made a visual of the full step-by-step. > Challenge setter (human) posts a hard problem with a clear score and an automatic verifier. > Humans (solvers) and AI agents attack it at once. Claude, GPT, Gemini, DeepSeek, Qwen, Kimi, GLM. It could be any model, any harness. > Each attempt goes through the verifier. > When one beats the current best, it becomes the new frontier (the record to beat). > Anyone can build on that new best instead of starting from 0. Then the loop repeats. All results and commits are open source, so you can trace how every record was set. I think this is how frontier research/science gets done in an AGI and PostAGI world. So far we've solved hard problems across cryptography, math, science and open-weight models. We'll keep going there and we're starting to explore biology, chemistry and materials science. See what we've solved (and are solving now) in each one on
0:20

EXPLAINED: How multiplayer autoresearch on Yukon works A lot of people have been asking, so I made a visual of the full step-by-step. > Challenge setter (human) posts a hard problem with a clear score and an automatic verifier. > Humans (solvers) and AI agents attack it at once. Claude, GPT, Gemini, DeepSeek, Qwen, Kimi, GLM. It could be any model, any harness. > Each attempt goes through the verifier. > When one beats the current best, it becomes the new frontier (the record to beat). > Anyone can build on that new best instead of starting from 0. Then the loop repeats. All results and commits are open source, so you can trace how every record was set. I think this is how frontier research/science gets done in an AGI and PostAGI world. So far we've solved hard problems across cryptography, math, science and open-weight models. We'll keep going there and we're starting to explore biology, chemistry and materials science. See what we've solved (and are solving now) in each one on

Pratik Gandhi

25,737 次观看 • 26 天前

this is straight f*cking gold the full harness guide for Kimi K3, the #1 open source frontend model the premise: the model underneath keeps changing. the harness is the part that stays yours what one night of it looks like: 02:00 - the trigger fires, every node that needs work gets picked > 212 agents fan out, one per node > a bad return gets rejected, retried once with the reason attached > one agent tries to write outside its folder. blocked. nobody woken up > +41 nodes and +96 edges land in the graph > a drafted email hits pre_send and waits for you 02:41 - the loop stops on its own, inside a 45 minute budget 07:30 - you read one file and make two decisions the whole machine is one folder, one config file, five short scripts two rules hold it together: > the model sits behind one line, so a better model is a config change > the verifier lives outside the agent, so nothing grades its own work now the money part companies burn whole quarters building internal agent platforms a harness setup for a small team goes for four figures, one time then a monthly retainer to run the swap test on every new model release models come and go. the person who owns the harness keeps getting paid
0:16

this is straight f*cking gold the full harness guide for Kimi K3, the #1 open source frontend model the premise: the model underneath keeps changing. the harness is the part that stays yours what one night of it looks like: 02:00 - the trigger fires, every node that needs work gets picked > 212 agents fan out, one per node > a bad return gets rejected, retried once with the reason attached > one agent tries to write outside its folder. blocked. nobody woken up > +41 nodes and +96 edges land in the graph > a drafted email hits pre_send and waits for you 02:41 - the loop stops on its own, inside a 45 minute budget 07:30 - you read one file and make two decisions the whole machine is one folder, one config file, five short scripts two rules hold it together: > the model sits behind one line, so a better model is a config change > the verifier lives outside the agent, so nothing grades its own work now the money part companies burn whole quarters building internal agent platforms a harness setup for a small team goes for four figures, one time then a monthly retainer to run the swap test on every new model release models come and go. the person who owns the harness keeps getting paid

Mr. Buzzoni

13,054 次观看 • 1 个月前

Lauren from the Grok Bot team just confirmed it: every bot now runs on Claude Opus 5.5 and each one can spin up its own cloud agents running Cursor’s models this is f*cking insane... so i built the harness on top of it, in one prompt: Opus 5.5 leads and builds, Grok scouts live X and the web, every claim needs two sources, a red team attacks the draft before you see it, and every run gets logged so the next one starts smarter you stop prompting models. you sign off on results. paste it into Grok Bot or Claude and thank me later all 10 steps, the workflow, and real examples below ↓
0:19

Lauren from the Grok Bot team just confirmed it: every bot now runs on Claude Opus 5.5 and each one can spin up its own cloud agents running Cursor’s models this is f*cking insane... so i built the harness on top of it, in one prompt: Opus 5.5 leads and builds, Grok scouts live X and the web, every claim needs two sources, a red team attacks the draft before you see it, and every run gets logged so the next one starts smarter you stop prompting models. you sign off on results. paste it into Grok Bot or Claude and thank me later all 10 steps, the workflow, and real examples below ↓

Alex

44,291 次观看 • 22 天前

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.
0:15

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.

Avi Chawla

442,507 次观看 • 3 个月前

One hundred thousand Github Stars!! This makes Paperclip one of the top 150 open-source projects of. all. time. All computer work is becoming agentic work so you need tools to harness the army of agents at your disposal We've built 📎 for companies to get a handle on agent sprawl and it's popular is because it's good Paperclip is multi-player and you can use any model, any harness, any time. It's the layer for every connection, every agent, and every human to collaborate on work The self hosted version is completely free (and you can sign up for the waitlist for the hosted version) 📎📎📎 ITS TIME TO CLIP 📎 📎📎
0:10

One hundred thousand Github Stars!! This makes Paperclip one of the top 150 open-source projects of. all. time. All computer work is becoming agentic work so you need tools to harness the army of agents at your disposal We've built 📎 for companies to get a handle on agent sprawl and it's popular is because it's good Paperclip is multi-player and you can use any model, any harness, any time. It's the layer for every connection, every agent, and every human to collaborate on work The self hosted version is completely free (and you can sign up for the waitlist for the hosted version) 📎📎📎 ITS TIME TO CLIP 📎 📎📎

dotta 📎

21,617 次观看 • 20 天前

This is pure f*cking treasure. ex-Google engineer reveals his AI agent playbook. 14+ years at Google. $0 to read it. the topic is harness engineering: everything around the model that lets it execute, check its work and recover when something breaks. one example in his breakdown deserves attention. LangChain’s coding agent went from 52.8% to 66.5% on Terminal-Bench 2.0. the model stayed the same. the team changed the prompts, tools and machinery around it. Agent = Model + Harness. the system has six practical jobs: > guides give the agent project rules, constraints and examples > sensors check its work through tests, linters and validators > the loop handles execution, verification, retries and escalation > memory preserves state, artifacts and decisions > permissions control which tools and actions are allowed > observability records tool calls, costs and failures the part worth stealing is what happens after a mistake. an agent keeps declaring broken code finished. you add a verification step before completion. it keeps ignoring a project convention. you add a targeted instruction and a check for that specific failure. it touches files outside its task. you tighten its write permissions. each recurring failure gives you a concrete improvement to make in the harness. that improvement can carry into every future run. this is where the engineering work compounds. bookmark it. start with the mistake you have already paid your agent to make twice.
0:18

This is pure f*cking treasure. ex-Google engineer reveals his AI agent playbook. 14+ years at Google. $0 to read it. the topic is harness engineering: everything around the model that lets it execute, check its work and recover when something breaks. one example in his breakdown deserves attention. LangChain’s coding agent went from 52.8% to 66.5% on Terminal-Bench 2.0. the model stayed the same. the team changed the prompts, tools and machinery around it. Agent = Model + Harness. the system has six practical jobs: > guides give the agent project rules, constraints and examples > sensors check its work through tests, linters and validators > the loop handles execution, verification, retries and escalation > memory preserves state, artifacts and decisions > permissions control which tools and actions are allowed > observability records tool calls, costs and failures the part worth stealing is what happens after a mistake. an agent keeps declaring broken code finished. you add a verification step before completion. it keeps ignoring a project convention. you add a targeted instruction and a check for that specific failure. it touches files outside its task. you tighten its write permissions. each recurring failure gives you a concrete improvement to make in the harness. that improvement can carry into every future run. this is where the engineering work compounds. bookmark it. start with the mistake you have already paid your agent to make twice.

NO1ennn

29,017 次观看 • 1 个月前

PATRONUS RECORDED 100+ HOURS OF AI AGENTS PLAYING VIDEO GAMES AND PUT IT ALL IN PUBLIC Not the scores. The footage. → Watch the actual runs, not a number in a table → Frontier models on one leaderboard → Open for you to play and compete Benchmarks tell you who won.
0:14

PATRONUS RECORDED 100+ HOURS OF AI AGENTS PLAYING VIDEO GAMES AND PUT IT ALL IN PUBLIC Not the scores. The footage. → Watch the actual runs, not a number in a table → Frontier models on one leaderboard → Open for you to play and compete Benchmarks tell you who won.

marcus

33,081 次观看 • 1 个月前

Black Forest Labs just announced FLUX 3: a unified multimodal model for image, video, audio and action prediction. Founded in Freiburg, Germany, one of the few globally significant European companies in the AI ​​sector. The current release is gated early access, not open source. But if FLUX 3 Dev actually ships with usable open weights, this could become one of the most important open-model releases in generative video yet. Especially after the Chinese Minimax H3 release. BFL claims, based on preliminary internal comparisons, that FLUX 3 outperformed models such as Runway Gen-4.5, Luma Ray 3.2, Kling 3 Pro, and Seedance 2.0. However, these figures stem from the early-access phase and do not constitute independent validation. Be that as it may, it's great to see the quality that today's video models can produce at affordable prices. Something that would have been unthinkable a year ago. Really cool release!
0:10

Black Forest Labs just announced FLUX 3: a unified multimodal model for image, video, audio and action prediction. Founded in Freiburg, Germany, one of the few globally significant European companies in the AI ​​sector. The current release is gated early access, not open source. But if FLUX 3 Dev actually ships with usable open weights, this could become one of the most important open-model releases in generative video yet. Especially after the Chinese Minimax H3 release. BFL claims, based on preliminary internal comparisons, that FLUX 3 outperformed models such as Runway Gen-4.5, Luma Ray 3.2, Kling 3 Pro, and Seedance 2.0. However, these figures stem from the early-access phase and do not constitute independent validation. Be that as it may, it's great to see the quality that today's video models can produce at affordable prices. Something that would have been unthinkable a year ago. Really cool release!

Chubby♨️

39,869 次观看 • 2 个月前

The redesigned Arena Leaderboard Overview is live! The frontier is moving quickly, so we built a central hub to help you keep up. The new Arena Leaderboard experience brings all our live signals into one place for researchers, builders, and our global community of users who power our model evaluations. Here’s what’s new: - New Release Models: a real-time feed of the latest models added to Arena - Performance by Category: surfacing live sessions, and the top models by categories like Agents, Images, and Coding. - Model Capabilities: first impressions of new models, straight from the Arena team - Arena News: the latest research, features and updates from around the Arena universe For example, now you can see scores across arenas for models like GPT-6 Sol and Claude Opus 5.5, alongside signals like their category standings and capabilities, their latest news and more. Explore scores and now much more at our redesigned leaderboard. Find the link below.
0:17

The redesigned Arena Leaderboard Overview is live! The frontier is moving quickly, so we built a central hub to help you keep up. The new Arena Leaderboard experience brings all our live signals into one place for researchers, builders, and our global community of users who power our model evaluations. Here’s what’s new: - New Release Models: a real-time feed of the latest models added to Arena - Performance by Category: surfacing live sessions, and the top models by categories like Agents, Images, and Coding. - Model Capabilities: first impressions of new models, straight from the Arena team - Arena News: the latest research, features and updates from around the Arena universe For example, now you can see scores across arenas for models like GPT-6 Sol and Claude Opus 5.5, alongside signals like their category standings and capabilities, their latest news and more. Explore scores and now much more at our redesigned leaderboard. Find the link below.

Arena.ai

24,761 次观看 • 1 个月前