Introducing agent-desktop Computer use for your agents ▲ E2E test desktop apps with agents ▲ Reads apps via the accessibility tree ▲ Drills into part of a window to save tokens ▲ Runs many agents on one app at once ▲ Works on macOS, Windows & Linux npm i... -g agent-desktopshow more

1/ Skeleton based drilling Agents don't need to see the whole screen, instead of exposing the entire screen, they can now drill through into only what they want.

2/ Multi-agent Run several agents on one desktop. Start a session with --multi-agent and give each agent an --agent-id. Each one gets its own cursor and label, so you can see who is doing what.

3/ Headless and headed Headless by default with an exception to few commands - `drag, menus, surfaces`

4/ With agent-browser agent-desktop pairs with agent-browser. agent-browser drives the web. agent-desktop drives native apps. Together your agent gets reliable computer use across the browser and the desktop.

Your agents can run agent-desktop in a Vercel Sandbox too. Start a Linux desktop in the cloud, run agent-desktop inside it, and get JSON back.

A ▲ Labs Experiment. Give your agents hands today npm i -g agent-desktop

shoutout to @vercel team for making agent-browser

reading apps through the accessibility tree is the smart move. way more reliable than guessing from screenshots

I built helix, a personal tool to manage my AI agents, so I’m curious: what happens if two agents try to change the same app state at once?

Yes sirrrrrr

lessgoo

For those of you who might be using @omarchy tell Claude to "fork the repo and start installing and testing. Then every time you runs into an error or a limitation, code a solution and retest. Keep testing until the entire stack works" Thank me later.

Tried it with Zed and it fumbled working in the background. MacOS Harness works better...

Was it taking the focus?

It was at first and in the background it couldn't type into Zed since Zed exposes no accessibility ref for the editor while MacOS Harness sends keys straight to Zed's process. I had told it "open Zed and type out a joke then save it as a file."

'osascript' which I believe is what macos harness does, we should explore this. Thanks for the feedback

The window drill-down is a useful detail. Would love to see full-window vs drilled-down token counts at the same task completion rate, including screenshot fallbacks when controls are missing from the accessibility tree.

There is focus problems on other frameworks using the accessibility tree any progress on this end?

was thinking about to build something like that, awesome work

Epic! There goes my whole weekend. LFG

Hoping for hyperland support soon

nice work

Thanks

this is sick. what does it do on apps with half-labeled accessibility trees, fall back to screenshots?

It's like a iframe issue in web. Doesn't fallback automatically but you can get it working using --headed It's a really good surface to work on. Getting it working for poor accessible elements would be a banger

ah got it, the iframe analogy helps. pretty cool shit you're working on. getting it to handle poorly labeled elements would be huge.

🫡

This should be good

great! will it works with agent-browser? Meaning it could for eg operate the browser and other apps together with an unified interface?

Yes! agent-desktop -> desktop apps, agent-browser -> browser surfaces / chromium apps

nice, thanks!

Drilling into just the place card (45 refs vs 75 for the whole window) is the part I want for long desktop runs. Couldn't try it on our headless Linux box yet since there's no AT-SPI bus there

👀

huge

Running many agents on one app makes state isolation part of the test design. If one changes the screen while another asserts against it, a failure may reflect the test setup rather than an app bug.

Great point!

@bot check this out

Background input looks useful for shared browser workflows. How do you handle a person taking over while an agent has actions queued? I'd want an explicit pause/hand-back signal before those actions resume, especially around logins and forms.

stop moving the mouse lol. use a grid

yeah reading the accessibility tree instead of guessing pixels is overdue. curious how often screenshot loops still win once the UI gets weird.

i'm tired of javascript boss

Bro how does it compare to Cua Driver?

What is this sorcery?!

Wowww

nice man nice. how does it do with linux? I will have to try it out. I was doing something similar with boxcar: but I have not worked on it in a bit. Linux is a little trickier bc of wayland and an inconsistency in accessibility trees in app distributions.

Wow, very cool! Looking forward to giving it a spin! Question: how does it handle permissions? E.g. How can I instruct it to never send e-mail or complete forms on websites?

@ctatedev Really exciting stuff!

accessibility tree instead of screenshots should save a ton of tokens

Can it play an Unreal game for me? asking for my hospital game