Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

At Uber, our testing agent hit an outage in Australia that I still think about. We had the recording. That's about it. No network trace for the run. No logs mapped to each step. No perf data. Just a video of an agent pressing a button and a team...

42,149 görüntüleme • 1 ay önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Stanford researchers did it again. They just built the agent-native version of Git. When an agent works on a longer task, the run builds up a lot of state. This includes files edited/created, a dev server, a database, installed packages, KV cache, etc. Say the agent is at step 10 and makes a mistake, maybe it misreads a traceback and rewrites a file that was actually fine. The tests start failing, and the run goes off track, although everything through step eight was correct. By default, the agent just tries to fix it, which creates more edits and tool calls. This burns more tokens and grows the context. The other options are a person stepping in to redirect it or restarting the whole run from step one. That's wasteful, because it pays for every model/tool call again and re-prefills the context. Moreover, since an agent's run is non-deterministic, it doesn't reproduce the same early steps anyway. The reason it's hard to just jump back exactly to a previous correct step and resume from there is that the trajectory is only a message log. It records what the agent said and which tools it called, but not the live state underneath. That state includes things like memory, open file handles, child processes, installed packages, /tmp, and KV cache. None of that is in the log. Git can version the files, but it doesn't snapshot the running process or the KV cache. Checking out step eight moves the files back, but the process is still sitting in step-ten memory with a cold cache. Shepherd is a runtime layer by Stanford that records the run as a trace of typed events rather than a flat log. Each agent-environment interaction becomes a commit, similar to Git, but it tracks the live run. Its commit includes the agent process and the filesystem together, copy-on-write, so a branch carries the actual state and not just the files. Going back to a previous step is then a single call that forks from that commit and continues from the exact state. The copy-on-write fork is roughly five times faster than docker commit, and because the prompt prefix through step eight is unchanged, the KV cache is reused over 95% on replay, so early steps aren't reprocessed again. Once the run can be forked, a meta-agent can sit on top and operate it. It watches the trace and reverts as soon as it looks wrong, before the bad write is committed. In practice, it's just Python calling fork, replay, and revert on the trace, rather than a separate control plane wired into the harness. Not everything is reversible though. Files and sandbox changes undo themselves, but a database write has no automatic undo, so it needs a matching undo step set up in advance. Something external, like a sent email or a real charge, can't be undone, so the supervisor's job there is to catch it before it fires. They tested this on a few public benchmarks. On CooperBench, where two agents work on the same codebase, adding a live supervisor took the pair-coding pass rate from 28.8% to 54.7%. It's still early and labeled alpha. The benefit mostly shows up when a run gets branched a lot over a heavy sandbox state, which is exactly where restarting wastes the most tokens and time. If Git was made to make file changes reversible, Shepherd is trying to do the same thing for a live agent run. Shepherd Repo: (don't forget to star it ⭐ ) That said, Shepherd reverts a bad step inside a run. The harness around it, the prompts, tools, and checks the supervisor relies on, still drifts across runs as models and dependencies change. Akshay wrote about making that harness repair itself, where a failing trace gets diagnosed, the fix is verified against the exact input that failed, and the failure is locked as a regression test so it can't recur. Read it below.

Avi Chawla

440,706 görüntüleme • 1 ay önce

I just built a Meta Ads diagnostic in Claude Code that tells you WHY your account broke, not just what changed 🤯 It spins up a team of agents that each investigate a different reason performance dropped, then argue against each other to kill the wrong answer before it ever reaches you. All inside Claude Code. Perfect for DTC brands and agencies who panic-kill creative the second CPA spikes. If you've watched ROAS fall off a cliff and opened Ads Manager with ten tabs going, you already know what happens next. Your gut says "creative fatigue." You kill your best-performing ad. A week later performance is still broken, because that was never the problem. Guessing wrong is the most expensive move in paid social. This workflow ends the guessing: → One agent investigates each competing theory — creative fatigue, budget and delivery changes, traffic quality, offer and seasonality → Each one is blind to the others, reasoning only from its own slice of the data so they can't bias each other → A refuter agent then attacks every surviving theory and tries to kill it → A theory only stands if the data can't disprove it → You get a ranked diagnosis: the real cause, the evidence for and against it, and the one move to make this week No anchoring on the first obvious answer. No killing winning creative on a hunch. No "here's what happened" reports that never tell you why. What you get: → Every theory tested in parallel instead of one biased guess → An adversarial pass that kills the wrong answer before you act on it → A ranked diagnosis with confidence levels and evidence both ways → A reusable workflow you drop next month's export into and re-run Built 100% in Claude Code with the new dynamic workflows. The first account I ran it on looked like textbook creative fatigue. The workflow disagreed, and traced the real cause to a budget change that had doubled spend and flooded delivery with junk traffic. I put together a full playbook with the exact workflow, the prompt, and how to run it on your own account. Want it for free? > Like this post > Comment "META" And I'll send it over (must be following so I can DM)

Mike Futia

12,772 görüntüleme • 2 ay önce

Today America roots for Norway, and I need you to understand exactly why. We love an underdog. It is not a preference. It is a genetic condition. Two hundred and fifty-three years ago, a handful of colonials with no navy, no army, and no chance decided to throw a tea party in Boston Harbor to spite the largest empire the world had ever seen. An empire the sun never set on. An empire that could have ended us in an afternoon. And when the greatest military on earth said you cannot possibly do that, our answer was, and remains, hold my beer. Then we picked a fight we had no business winning. Then we declared ourselves free before we had actually won anything. Then we went out and made it true. We turned 250 last week. Still here. Still loud. That is the whole American operating system. The impossible odds. The comeback. The redemption. We do not just like that story, we are that story, and we will root for it anywhere we find it. It is why a bunch of college kids beating a Soviet hockey machine in 1980 still makes grown men cry in bars. So today. Norway. Five and a half million people. A country with more sheep than starting lineups. Playing England, a founding nation of the sport, on the biggest stage there is. Now let me be fair. We love England. Truly. Best ally we have got, and I say that with a full heart. Though the Japanese are gaining fast, so do not get comfortable. We love the Scots. We love the Northern Irish. We love our drunk cousins in Ireland, and yes I am lumping you all in, because half of you moved here anyway and you hate the English for reasons we are all familiar with. Nobody loves Wales. Nobody knows Wales exists. Nobody can understand a word of it. But I digress. Because today is not about any of that. Today is about five million Vikings walking into a stadium against an empire, with everyone on earth telling them they cannot possibly do this. You know exactly what we say to that. Let’s row!!! NORWAY!!!!!!

Selene Mariposa

71,271 görüntüleme • 1 ay önce

I had the same thought so I've been playing with it in nanochat. E.g. here's 8 agents (4 claude, 4 codex), with 1 GPU each running nanochat experiments (trying to delete logit softcap without regression). The TLDR is that it doesn't work and it's a mess... but it's still very pretty to look at :) I tried a few setups: 8 independent solo researchers, 1 chief scientist giving work to 8 junior researchers, etc. Each research program is a git branch, each scientist forks it into a feature branch, git worktrees for isolation, simple files for comms, skip Docker/VMs for simplicity atm (I find that instructions are enough to prevent interference). Research org runs in tmux window grids of interactive sessions (like Teams) so that it's pretty to look at, see their individual work, and "take over" if needed, i.e. no -p. But ok the reason it doesn't work so far is that the agents' ideas are just pretty bad out of the box, even at highest intelligence. They don't think carefully though experiment design, they run a bit non-sensical variations, they don't create strong baselines and ablate things properly, they don't carefully control for runtime or flops. (just as an example, an agent yesterday "discovered" that increasing the hidden size of the network improves the validation loss, which is a totally spurious result given that a bigger network will have a lower validation loss in the infinite data regime, but then it also trains for a lot longer, it's not clear why I had to come in to point that out). They are very good at implementing any given well-scoped and described idea but they don't creatively generate them. But the goal is that you are now programming an organization (e.g. a "research org") and its individual agents, so the "source code" is the collection of prompts, skills, tools, etc. and processes that make it up. E.g. a daily standup in the morning is now part of the "org code". And optimizing nanochat pretraining is just one of the many tasks (almost like an eval). Then - given an arbitrary task, how quickly does your research org generate progress on it?

Andrej Karpathy

1,649,304 görüntüleme • 5 ay önce