Built a test suite for T3 Code with e2e ⚡ 40 tests, testing 6 coding agents (Codex, Claude, Grok, OpenCode, Pi, Antigravity) mocked at CLI level It already caught a small bug: Stop doesn't stop a turn that's still starting

it's using btw @jullerino - happy to create a PR with this!

👀

here's a PR:

the real question here is can my mac handle this

I dont think you need that much complexity to get this bug like its there if you have used t3 code buddy !

to be fair that bug is known to everyone who has used t3 for more than an hour or so but still super impressive

@kacperkapusciak Took me, longer than it should have to understand. It’s a test suite for the T3 codebase, right?

@kacperkapusciak yes!

A Stop button missing the 'turn is still starting' state is such a classic race. Nice catch. Mocking all six agents at the CLI level sounds like the right way to find these.

40 tests written by AI that test nothing real

CLI-level mocks are like a flight simulator: they let you test the stop button and other control paths without depending on a live agent run.

i'd worry about a late event from the cancelled turn landing in the next one. does the suite cover stop, restart, then the old startup finally completing?

Mocking at the CLI level is the part I'd watch: when Codex or Claude change their output format, the mocks keep passing while the real integration breaks. Do you re-record the fixtures from the real CLIs on a schedule?

How can i contribute to this project?

we're accepting pull requests on GitHub:

Thanks

catching bugs like that is key for smoother interactions, great job!

Does stop need a turn id before it can cancel?

how do you mock them at the cli level, recorded output replayed by a fake binary? i'd guess the painful part is keeping those fixtures in sync each time one of the six changes its output format

That bug class shows up in every harness. Tests cover idle and running, but "starting" is its own state and Stop has to mean something there too. Mocking at the CLI level is a nice way to hit those in-between states on purpose.

Nice catch! Wondering how the suite is written tho: mostly agent.act with the replay cache, or deterministic steps? Asking bc I've been thinking about where the intent lives. Steps in UI language have to change in a lot of places when the UI changes. Steps in high-level language don't, but then the same path in different tests gets healed separately, right? With page objects that's one method and one diff in domain language. With the cache it's gitignored by default, and when committed you review a JSON of recorded actions per test. Are we supposed to commit .e2e/cache and review those diffs?

Which of the six was the hardest to mock cleanly at the CLI level? Streaming output tends to break these harnesses before tool-call interception does.

It doesn't have to be this hard, friend.

CLI level mocks for agent testing are where the real dev-velocity gains are. Most people are still trying to e2e test the UI.

Testing at the CLI boundary is the right call, that's where the weird state lives. "Stop doesn't stop a turn that's still starting" is exactly the bug users hit and nobody can repro. Are you running this on every agent CLI release?

Forty e2e tests across six coding agents, mocked at the CLI level, is a solid safety net for T3 Code. Nice that it already caught the Stop bug 👀

bro ilysm