正在加载视频...

视频加载失败

返回首页

New open-source agent harness just landed! I got early access to TrueForge by TrueFoundry and have been running it locally for the past few days. The harness layer deserves as much attention as the model, and open source matters here because you can inspect the loop, run it on...

omarsar0's profile picture

elvis

314,429 subscribers

11,202 次观看 • 1 天前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.
0:18

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and let an automated loop rewrite only the code around it. That code layer is the harness, the runtime wrapper that feeds the model context, runs its tool calls, and decides when a run ends. By the time the loop finished, the system had essentially matched Sonnet 4.6 on the benchmark's headline metric, at roughly 7x lower cost per task. Zero weights changed. The gain existed because of where the model was failing. The judge only grades files saved in the right place under the exact requested filename, and the model kept doing the legal analysis correctly, then saving it under the wrong name, dropping it in a scratch folder, or never writing it at all. So the 0% was never measuring legal reasoning. It was measuring the harness. Hand-tuning that layer is slow and model-specific, so they automated it. A Claude proposer adds exactly one mechanism per iteration, and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. What the loop discovered says a lot about where agents actually fail. → The biggest single gain was file handling, not intelligence. An automatic step that lands the deliverable exactly where the judge expects it beat every prompt change, with zero extra model tokens. → Code fixes transferred across models, prompt playbooks did not. The same harness lifted a smaller model from the same family by 14 points, but the tuned prompts hurt a different model family on tasks it could already finish. → The harness mattered more than anything else. Same model, same judge, same tasks, and five different harnesses scored anywhere between 3.5% and 80.1%. The gains do eventually flatten, and the remaining misses look like real capability gaps. At some point the wrapper runs out of tricks and the model has to carry the work. But the lesson holds. A benchmark score measures the model and its harness together, and until the harness is fixed, it's impossible to know which one failed. I highly recommend reading this: I also wrote a deep dive on agent harness engineering a while back, covering the orchestration loop, tools, memory, context management, and everything that turns a stateless LLM into a capable agent. The article is quoted below.

Akshay 🚀

244,567 次观看 • 1 个月前

let me save you 3 hours of head scratching. if you're running local models like Qwen3.5-35B-A3B through Claude Code via llama.cpp's Anthropic endpoint, the chain will break every 3 to 5 minutes. tool call fails. flow stops. you reprompt. it recovers. 2 minutes later it stops again. the model is fine. the harness chokes on local inference latency. switch to OpenCode. same localhost endpoint. same model. same GPU. the chain doesn't break. the tradeoff: OpenCode sometimes loops. the model forgets what it already read and repeats the same tool call. but a loop you can interrupt. a broken chain kills your momentum and you start over. watch both side by side. proprietary agent vs open source agent. same 3B model. different failure modes. pick your poison.
0:19

let me save you 3 hours of head scratching. if you're running local models like Qwen3.5-35B-A3B through Claude Code via llama.cpp's Anthropic endpoint, the chain will break every 3 to 5 minutes. tool call fails. flow stops. you reprompt. it recovers. 2 minutes later it stops again. the model is fine. the harness chokes on local inference latency. switch to OpenCode. same localhost endpoint. same model. same GPU. the chain doesn't break. the tradeoff: OpenCode sometimes loops. the model forgets what it already read and repeats the same tool call. but a loop you can interrupt. a broken chain kills your momentum and you start over. watch both side by side. proprietary agent vs open source agent. same 3B model. different failure modes. pick your poison.

Sudo su

72,538 次观看 • 5 个月前

Big win for open-source LLMs! DeepSeek V4 Pro holds the top open-weights score on SWE-bench Verified, in the GPT-5.5 range. GLM 5.2 leads the open-weight intelligence index and sits near the closed frontier on long-horizon coding. But this leaderboard number is a weak proxy for real performance. It comes from one task set, run through one harness, served at one precision. The same weights can even score differently across providers, since many hosts quantize activations to fp8 and drift the model off its reference weights. Real performance is determined based on whether a model can read a repo, make coordinated edits across files, run the tests, and recover when one breaks. By that measure, the top open models hold up, but only inside the right harness. The teams that actually put DeepSeek V4 into production pipelines as a frontier substitute got there through the harness they built around the model, not by picking a stronger model. If you want to see this in practice, Cline (64k+ stars) has actually built that harness around open models, tuned so they run at production quality. And it's tuned so that these LLMs can run at production quality, with plan and act modes, checkpoints, and terminal feedback. ClinePass is the new access layer on top of it. It runs a curated set of those models inside Cline, narrowed to the ones tested for coding-agent use, with 2 to 5x the standard rate limits and no separate provider accounts, keys, or billing to track. The video below shows the setup, and I worked with the team to put this together. It runs alongside custom keys and local models as well, not in place of them.
0:14

Big win for open-source LLMs! DeepSeek V4 Pro holds the top open-weights score on SWE-bench Verified, in the GPT-5.5 range. GLM 5.2 leads the open-weight intelligence index and sits near the closed frontier on long-horizon coding. But this leaderboard number is a weak proxy for real performance. It comes from one task set, run through one harness, served at one precision. The same weights can even score differently across providers, since many hosts quantize activations to fp8 and drift the model off its reference weights. Real performance is determined based on whether a model can read a repo, make coordinated edits across files, run the tests, and recover when one breaks. By that measure, the top open models hold up, but only inside the right harness. The teams that actually put DeepSeek V4 into production pipelines as a frontier substitute got there through the harness they built around the model, not by picking a stronger model. If you want to see this in practice, Cline (64k+ stars) has actually built that harness around open models, tuned so they run at production quality. And it's tuned so that these LLMs can run at production quality, with plan and act modes, checkpoints, and terminal feedback. ClinePass is the new access layer on top of it. It runs a curated set of those models inside Cline, narrowed to the ones tested for coding-agent use, with 2 to 5x the standard rate limits and no separate provider accounts, keys, or billing to track. The video below shows the setup, and I worked with the team to put this together. It runs alongside custom keys and local models as well, not in place of them.

Avi Chawla

44,124 次观看 • 1 个月前

You don’t have to use one model (or one provider!) for everything. With Workshop, you can combine frontier and local models in the same workflow. For example: Opus can be the main agent, and delegate specific tasks to Gemma 4 via subagents. Better quality where it matters. Better privacy, speed, and cost where it counts. One workflow, best model for each task.
0:18

You don’t have to use one model (or one provider!) for everything. With Workshop, you can combine frontier and local models in the same workflow. For example: Opus can be the main agent, and delegate specific tasks to Gemma 4 via subagents. Better quality where it matters. Better privacy, speed, and cost where it counts. One workflow, best model for each task.

Workshop AI

904,641 次观看 • 4 个月前

Okay... this is actually insane. OpenCodex feels like the open-source breakthrough I've been waiting for. The best part : You can plug multiple providers into the same OpenAI Codex harness and switch between models depending on the task. Running low on tokens? No problem. Use another provider. OpenRouter free model today? Plug it in. This completely changes how I think about AI coding workflows. And yes... it even works on mobile. OpenCodex might be one of the most useful open-source AI projects I've seen this year. OpenAI built an incredible harness. The open-source community just made it universal.
0:10

Okay... this is actually insane. OpenCodex feels like the open-source breakthrough I've been waiting for. The best part : You can plug multiple providers into the same OpenAI Codex harness and switch between models depending on the task. Running low on tokens? No problem. Use another provider. OpenRouter free model today? Plug it in. This completely changes how I think about AI coding workflows. And yes... it even works on mobile. OpenCodex might be one of the most useful open-source AI projects I've seen this year. OpenAI built an incredible harness. The open-source community just made it universal.

CHOI

42,894 次观看 • 1 个月前

we just released a new blog "Training a coding agent using the OpenCode harness in remote HF sandboxes with TRL and OpenEnv" you can take a real coding agent (OpenCode), let it run its own tool loop against real coding problems, and train it with RL on the exact tokens it produced and every rollout runs in its own remote HF sandbox, so rollouts scale out beyond one machine the loop: - OpenCode owns its tool loop inside an OpenEnv sandbox - an in-sandbox proxy records the real token ids + logprobs, per turn - a hidden-test verifier scores the result, and that is the reward - TRL trains with AsyncGRPO, weights sync back to vLLM over NCCL blog + runnable example:
0:13

we just released a new blog "Training a coding agent using the OpenCode harness in remote HF sandboxes with TRL and OpenEnv" you can take a real coding agent (OpenCode), let it run its own tool loop against real coding problems, and train it with RL on the exact tokens it produced and every rollout runs in its own remote HF sandbox, so rollouts scale out beyond one machine the loop: - OpenCode owns its tool loop inside an OpenEnv sandbox - an in-sandbox proxy records the real token ids + logprobs, per turn - a hidden-test verifier scores the result, and that is the reward - TRL trains with AsyncGRPO, weights sync back to vLLM over NCCL blog + runnable example:

Sergio Paniego

36,159 次观看 • 15 天前

ANTHROPIC JUST TURNED AI AGENTS INTO GIT REPOS Anthropic shipped "ant" - a CLI that runs every Claude API endpoint straight from your terminal. The headline isn't the terminal access. It's that you can now version-control an AI agent as YAML in Git and have CI sync it to the Claude Platform, the same way you ship code. - Every API resource is a subcommand: messages, models, files, agents, sessions - Define an agent in a YAML file, check it into your repo, and keep it in sync with one update command - Spin up a session, send it an event, then pull every event and tool call back from the same CLI - Claude Code knows how to drive ant out of the box - it shells out and reads the results with no glue code Agents just stopped being prompts you babysit and became infrastructure you deploy.
0:18

ANTHROPIC JUST TURNED AI AGENTS INTO GIT REPOS Anthropic shipped "ant" - a CLI that runs every Claude API endpoint straight from your terminal. The headline isn't the terminal access. It's that you can now version-control an AI agent as YAML in Git and have CI sync it to the Claude Platform, the same way you ship code. - Every API resource is a subcommand: messages, models, files, agents, sessions - Define an agent in a YAML file, check it into your repo, and keep it in sync with one update command - Spin up a session, send it an event, then pull every event and tool call back from the same CLI - Claude Code knows how to drive ant out of the box - it shells out and reads the results with no glue code Agents just stopped being prompts you babysit and became infrastructure you deploy.

BuBBliK

200,402 次观看 • 2 个月前

subagents are just recursive agents where you can apply different prompts + models depending on the task. since they’re just a primitive, Cursor cli can actually spawn subagents by calling cursor-agent in headless mode via shell commands. that’s what makes the cli so nice. you can extend it, experiment, and have a lot of fun exploring orchestration patterns. here’s one way to do it w. dynamic model selection: 1. create a subagents.mdc rule 2. drop in: ``` --- alwaysApply: true --- ALWAYS spawn subagents by running `cursor-agent -p [task] --output-format=text --force --model [model]` in the terminal. Each subagent should return a summary of the changes it made. Subagents should be used for ALL tasks You can adopt a fan-out pattern where you spawn subagents to perform parallel isolated tasks, and then fan-in the results. Use the following models: - `--model gpt-5` for reasoning, researching, and planning - `--model sonnet-4` for implementation ``` 3. start cursor cli and try it out you can also adjust the rule to be more explicit when it should use subagents, when not to, which models when etc.
0:15

subagents are just recursive agents where you can apply different prompts + models depending on the task. since they’re just a primitive, Cursor cli can actually spawn subagents by calling cursor-agent in headless mode via shell commands. that’s what makes the cli so nice. you can extend it, experiment, and have a lot of fun exploring orchestration patterns. here’s one way to do it w. dynamic model selection: 1. create a subagents.mdc rule 2. drop in: ``` --- alwaysApply: true --- ALWAYS spawn subagents by running `cursor-agent -p [task] --output-format=text --force --model [model]` in the terminal. Each subagent should return a summary of the changes it made. Subagents should be used for ALL tasks You can adopt a fan-out pattern where you spawn subagents to perform parallel isolated tasks, and then fan-in the results. Use the following models: - `--model gpt-5` for reasoning, researching, and planning - `--model sonnet-4` for implementation ``` 3. start cursor cli and try it out you can also adjust the rule to be more explicit when it should use subagents, when not to, which models when etc.

eric zakariasson

57,554 次观看 • 1 年前

whoever leaked this has bigger balls than sense Google Research and MIT ran the same agent jobs 260 different ways for Nature last month: they held the prompts, the tools and the compute budget identical and moved nothing but the wiring between the agents, and the same work swung from 70% worse than a single agent to 80.8% better, averaging out at 0.0% i ran my own single agent against the task list first and it cleared 6 of 10 alone, already past the line where a crew starts subtracting this is Graph Engineering, the layer that decides whether a crew is worth 80% more or 70% less, and it installs into the agent you already pay for: - score your solo agent on the real task first: above roughly 45% success that study predicts zero to negative returns from any crew you put around it - under that line, put one supervisor over the fan out: crews with no correction step amplified their own errors to 17.2x the single agent rate, supervised aggregation held it to 4.4x - give every worker one output and let none of them read a peer's draft, so a wrong step reaches the supervisor instead of four other agents - run the comparison again after every model upgrade, because a better model raises your baseline and a higher baseline is what makes a crew stop paying - keep the single agent alive as the control, the only number that says the wiring is earning its calls turns out the shape does not travel: the biggest win came off a finance task under one supervisor and the worst collapse off a planning task with independent agents my position, and it is the arguable one: a crew is a bet on your own diagram, and the model you pick moves that bet less than one arrow does bookmark this, the three moves that draw those arrows before you pay for one extra call are in the post below ↓
0:15

whoever leaked this has bigger balls than sense Google Research and MIT ran the same agent jobs 260 different ways for Nature last month: they held the prompts, the tools and the compute budget identical and moved nothing but the wiring between the agents, and the same work swung from 70% worse than a single agent to 80.8% better, averaging out at 0.0% i ran my own single agent against the task list first and it cleared 6 of 10 alone, already past the line where a crew starts subtracting this is Graph Engineering, the layer that decides whether a crew is worth 80% more or 70% less, and it installs into the agent you already pay for: - score your solo agent on the real task first: above roughly 45% success that study predicts zero to negative returns from any crew you put around it - under that line, put one supervisor over the fan out: crews with no correction step amplified their own errors to 17.2x the single agent rate, supervised aggregation held it to 4.4x - give every worker one output and let none of them read a peer's draft, so a wrong step reaches the supervisor instead of four other agents - run the comparison again after every model upgrade, because a better model raises your baseline and a higher baseline is what makes a crew stop paying - keep the single agent alive as the control, the only number that says the wiring is earning its calls turns out the shape does not travel: the biggest win came off a finance task under one supervisor and the worst collapse off a planning task with independent agents my position, and it is the arguable one: a crew is a bet on your own diagram, and the model you pick moves that bet less than one arrow does bookmark this, the three moves that draw those arrows before you pay for one extra call are in the post below ↓

Argona

887,542 次观看 • 6 天前

Let me explain the agent loop, simple It's the core of every agentic system, and the part most people overcomplicate It's just this: 1. Send messages to the model 2. Model responds, maybe calls a tool 3. You run the tool 4. Append the result back to messages 5. Repeat until stop_reason is end_turn Step 4 is the whole thing, the write-back is what makes it an agent The model has to see what actually happened before it decides the next move That's the entire loop... understand this cold before you reach for a framework
0:14

Let me explain the agent loop, simple It's the core of every agentic system, and the part most people overcomplicate It's just this: 1. Send messages to the model 2. Model responds, maybe calls a tool 3. You run the tool 4. Append the result back to messages 5. Repeat until stop_reason is end_turn Step 4 is the whole thing, the write-back is what makes it an agent The model has to see what actually happened before it decides the next move That's the entire loop... understand this cold before you reach for a framework

Daniel San

12,514 次观看 • 2 个月前

Alright, now that we know *what* an agent is, how does it actually work? When you ask for help on a task, the agent plans a series of steps and executes them directly in the application on your behalf, using the tools it has access to. Say you are booking a local service or trying to organize your inbox (which typically takes multiple steps): the AI model first plans how to achieve the task using its existing knowledge and then interacts with your inbox to execute the task. The agent will continue until it is confident the task has been successfully completed.
0:15

Alright, now that we know *what* an agent is, how does it actually work? When you ask for help on a task, the agent plans a series of steps and executes them directly in the application on your behalf, using the tools it has access to. Say you are booking a local service or trying to organize your inbox (which typically takes multiple steps): the AI model first plans how to achieve the task using its existing knowledge and then interacts with your inbox to execute the task. The agent will continue until it is confident the task has been successfully completed.

Google AI

22,487 次观看 • 8 个月前

Most AI agent setups treat every message the same. Simple question? top-tier model. complex task? top-tier model. Your token bill just keeps climbing. I tested OpenSquilla this week on a real document drafting workflow, and the routing caught me off guard. It judges each message's complexity locally, then picks the model tier that fits. Simple tasks go to cheaper models. Complex ones still get the heavy lifting done. You're not paying reasoning tokens for a "hello." I ran a longer workflow, and the context didn't collapse the way it usually does. It distills important information before compression, so you're not starting from scratch mid-session. If you run agents regularly, the bill adds up faster than you think. This is built specifically for that problem. They're running the 10M Token Bill Challenge right now. worth joining if you want to see what smart routing actually saves you in practice. #10MTokenChallenge OpenSquilla
0:13

Most AI agent setups treat every message the same. Simple question? top-tier model. complex task? top-tier model. Your token bill just keeps climbing. I tested OpenSquilla this week on a real document drafting workflow, and the routing caught me off guard. It judges each message's complexity locally, then picks the model tier that fits. Simple tasks go to cheaper models. Complex ones still get the heavy lifting done. You're not paying reasoning tokens for a "hello." I ran a longer workflow, and the context didn't collapse the way it usually does. It distills important information before compression, so you're not starting from scratch mid-session. If you run agents regularly, the bill adds up faster than you think. This is built specifically for that problem. They're running the 10M Token Bill Challenge right now. worth joining if you want to see what smart routing actually saves you in practice. #10MTokenChallenge OpenSquilla

Parul Gautam

26,685 次观看 • 3 个月前

Dynamic workflows are a generalization of harnesses, automations, loops, routing, and graphs. It's the most powerful feature I have built into my agent orchestrator. Supports all kinds of patterns that leverage different agent backends (claude, codex, pi, hermes,...). It's a meta-harness approach that unlocks new forms of test-time compute. Example of use cases it supports: > LLM councils to get different perspectives from LLMs or plan more intensively > Dynamically routing tasks to different agents based on needs (e.g., cost efficiency and optimal intelligence) > Advisor/Judge + executor workflows and pretty much any complex graph-based pattern required by the task. I find it especially useful for long-running work and code reviewing. > Agent teams that talk to each other if needed for the task. I like to use this for AI editing, artifact creation, and other creative tasks. And I am sure it supports so many things that I haven't discovered yet. I got inspired by the dynamic workflow feature released by the Claude Code team. I had actually built it earlier this year but wanted to generalize it across different agent backends. I think this is going to become more popular in the coming days. I will share more of my findings soon.
0:11

Dynamic workflows are a generalization of harnesses, automations, loops, routing, and graphs. It's the most powerful feature I have built into my agent orchestrator. Supports all kinds of patterns that leverage different agent backends (claude, codex, pi, hermes,...). It's a meta-harness approach that unlocks new forms of test-time compute. Example of use cases it supports: > LLM councils to get different perspectives from LLMs or plan more intensively > Dynamically routing tasks to different agents based on needs (e.g., cost efficiency and optimal intelligence) > Advisor/Judge + executor workflows and pretty much any complex graph-based pattern required by the task. I find it especially useful for long-running work and code reviewing. > Agent teams that talk to each other if needed for the task. I like to use this for AI editing, artifact creation, and other creative tasks. And I am sure it supports so many things that I haven't discovered yet. I got inspired by the dynamic workflow feature released by the Claude Code team. I had actually built it earlier this year but wanted to generalize it across different agent backends. I think this is going to become more popular in the coming days. I will share more of my findings soon.

elvis

32,539 次观看 • 28 天前

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.
0:15

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.

Avi Chawla

440,706 次观看 • 1 个月前

Replit, Vercel, and OpenAI have built very cool agent-native applications, but nobody else has passed the demo stage. Building agents that work is complex. Teams aren't shipping agents because we don't have good tooling yet (and most of us don't know how to do this well.) A couple of days ago, the CopilotKit🪁 team announced a collaboration with . You can now use LangGraph with CoAgents to build agent-native applications, and here is everything you need to know about that: CoAgents is fully open-source, and you can use it to do the following: • Human-in-the-loop to steer and correct the agent • Stream intermediate agent state • Real-time state sharing between the agent and the application • Agentic generative UI to build trust that the agent is on the right path Start this GitHub Repository: Thanks to the team for giving me early access and collaborating with me on this post.
0:10

Replit, Vercel, and OpenAI have built very cool agent-native applications, but nobody else has passed the demo stage. Building agents that work is complex. Teams aren't shipping agents because we don't have good tooling yet (and most of us don't know how to do this well.) A couple of days ago, the CopilotKit🪁 team announced a collaboration with . You can now use LangGraph with CoAgents to build agent-native applications, and here is everything you need to know about that: CoAgents is fully open-source, and you can use it to do the following: • Human-in-the-loop to steer and correct the agent • Stream intermediate agent state • Real-time state sharing between the agent and the application • Agentic generative UI to build trust that the agent is on the right path Start this GitHub Repository: Thanks to the team for giving me early access and collaborating with me on this post.

Santiago

63,073 次观看 • 1 年前

MiniMax is the James Bond of AI agents. It uses the world's first open-weight model (MiniMax-M1), and it squeezes every bit of power from it. The agent takes a prompt and does more than any other agent in the market right now: 1. It can do Deep Research 2. It can write code 3. It can design web pages 4. It can build 3D models I built 5 different experiences using MiniMax and recorded them for you:
0:13

MiniMax is the James Bond of AI agents. It uses the world's first open-weight model (MiniMax-M1), and it squeezes every bit of power from it. The agent takes a prompt and does more than any other agent in the market right now: 1. It can do Deep Research 2. It can write code 3. It can design web pages 4. It can build 3D models I built 5 different experiences using MiniMax and recorded them for you:

Santiago

44,730 次观看 • 1 年前

this might be the E2B killer for AI agent sandboxes. forkd is an open-source microVM sandbox runtime built on Firecracker, made for AI agent fan-out, code interpreters, eval harnesses, anything that spins up a lot of short-lived sandboxes. before: every sandbox cold-boots its own VM and re-imports the whole runtime from scratch, numpy, torch, JIT compilation, model weights, all of it. now: forkd boots one parent VM once, warms it with your runtime already imported, then forks children from that snapshot using copy-on-write memory. the repo's own benchmark: spawning 100 sandboxes takes 101ms with forkd, versus 759ms for a raw Firecracker cold-boot, and well over a minute for Docker or gVisor. the SDK is a literal drop-in for E2B's Python client, so if you're already running code-interpreter agents on it, swapping the import line gets you a self-hosted runtime with the same isolation model at a fraction of the per-sandbox cost. pre-built recipes ship for e2b-style code interpreters, Jupyter kernels, SWE-bench coding agents, and Playwright browser fan-out
0:14

this might be the E2B killer for AI agent sandboxes. forkd is an open-source microVM sandbox runtime built on Firecracker, made for AI agent fan-out, code interpreters, eval harnesses, anything that spins up a lot of short-lived sandboxes. before: every sandbox cold-boots its own VM and re-imports the whole runtime from scratch, numpy, torch, JIT compilation, model weights, all of it. now: forkd boots one parent VM once, warms it with your runtime already imported, then forks children from that snapshot using copy-on-write memory. the repo's own benchmark: spawning 100 sandboxes takes 101ms with forkd, versus 759ms for a raw Firecracker cold-boot, and well over a minute for Docker or gVisor. the SDK is a literal drop-in for E2B's Python client, so if you're already running code-interpreter agents on it, swapping the import line gets you a self-hosted runtime with the same isolation model at a fraction of the per-sandbox cost. pre-built recipes ship for e2b-style code interpreters, Jupyter kernels, SWE-bench coding agents, and Playwright browser fan-out

Oliver Prompts

20,005 次观看 • 7 天前

50% cheaper Claude inference with just one line of code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.
0:12

50% cheaper Claude inference with just one line of code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.

Akshay 🚀

63,725 次观看 • 1 个月前

You can now post-train a model inside your existing production harness with our platform, AC2. A production harness is a whole engineered system around the LLM, with its own context management, tools, sandboxing, and control flows. Porting that into a new training runtime can be expensive and could introduce train-test mismatch, where the policy is optimized against a simulated harness and then struggles in production. All you need to do is swap out the harness’ LLM response endpoint to one provided by AC2, and expose a lightweight protocol for AC2 to initiate and grade rollouts; the trainer handles the rest.
0:13

You can now post-train a model inside your existing production harness with our platform, AC2. A production harness is a whole engineered system around the LLM, with its own context management, tools, sandboxing, and control flows. Porting that into a new training runtime can be expensive and could introduce train-test mismatch, where the policy is optimized against a simulated harness and then struggles in production. All you need to do is swap out the harness’ LLM response endpoint to one provided by AC2, and expose a lightweight protocol for AC2 to initiate and grade rollouts; the trainer handles the rest.

Applied Compute

180,495 次观看 • 13 天前