I made a computer use agent that works without screenshots. It uses the macOS accessibility API to interact with applications. You can run it in the cloud with Daytona, connect it to any LLM, and control any macOS application. Here it is building a simple program in Xcode:

@daytonaio Ok now combine this with @typesafeai jev pls

@daytonaio @typesafeai If you have access, we can do it!

@daytonaio this is sweet james! when do you come back?

@daytonaio Thanks! Hopefully in February. 🙂

@daytonaio how does it do with electron apps?

@daytonaio I only gave this agent tools for operating native macOS apps. Fortunately, there are Playwright tools for controlling Electron apps. For the most production ready agent, I would combine accessibility APIs, screenshots/vision and Playwright into one agent!

@daytonaio This looks really Nice!

@daytonaio this is nice but my experience with AX tree is that is unreliable sooner or later with many apps. which is point of view? plus there are a plenty of apps which don't expose the full tree, only some windwos and menus limitating what you can do

@daytonaio Can you give me some examples? This method has been working great for me but I need to test more. (I'm also not counting Electron apps since they can be controlled with Playwright.)

Don't take me wrong, it can work but it is not fully reliable. Apps can change their UI and the AX tree, this could broke your automations. I have experience even before Claude Code and gang with this matter and it tends to be fragile in real professional workflows. Plus, you have a plently of apps that don't expose the full AX tree. Take the Adobe Suite (ex. Premiere Pro), it exposes the window and the menus, but the work surface is drawn by their own toolkit, so timeline, program monitor and most panels come through as one opaque element or unlabeled groups. The tree tells you a panel exists, not that there's a clip in it, where it starts or what's selected. Honestly being able to really automate your apps in real professional workflows with computer use on your desktop needs still be fully solved

@daytonaio Accessibility tree instead of screenshots feels like the right trade for reliability. Curious how it handles custom-drawn controls that don't expose AX.

This prototype can only see AX! I've previously made some vision-based agents. They were relatively slow, expensive and prone to hallucination. One approach would be to extend this agent with a vision-based subagent, activated when needed and scoped to a region of the screen. Here's a related previous project I made:

@daytonaio how they task then tell me

@daytonaio GitHub?

@daytonaio