computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search... youtube for lofi beats", a big model figures it out once, does it, and saves the recipe as a json spec: {"op":"act","do":"open","target":"Safari"} {"op":"act","do":"type","text":"{query}"} every time after that, a tiny model on my laptop recognizes the command, fills in the parameters, and replays the recipe with CUA say it once, it's slow. say it again, it's instant. the fine tuned tiny model is 350 million parameters. its only job is recognizing which of my commands i just said. and when the big model does plan, it streams actions line by line. each line runs the moment it's finished, so the first action is executing while the model is still writing the second. everything it learns lives in one plain text file on my disk. i've been playing around a lot with Cua by Francesco and peekaboo by Peter Steinberger 🦞 . curious whether a standard computer use spec makes sense, one that streaming models can drive to make computer use actually fast.show more

this is what im playing around with

👀

:D

smells like there’s a big if true idea here, cool stuff

Finding out in realtime :D

the latency problem is less “the model is slow” and more “the model is being asked to rediscover the same procedure.” caching the routine path feels like the right boundary: keep novel decisions in the loop, move deterministic commands out of it.

It isnt quite transparent of how caching works with LLMs. This gives me more control over things. But yes I think its a similar concept that I’m implementing

yeah, cache hits are cute until you cannot see what it actually did.

they also dont really disclose cache costs,. atleast to my understanding

Good stuff, hope u can crack computer use with local models working with the cua team. Challenge - can you think of how to solve the problem of apps which don't surface accessibility trees such as most enterprise/business software? Or even apps running in VM's / remote desktops. Closest thing I can think of is using a harness which utilizes a thinking model & a model like Locate Anything 3B. But you are the expert here, hopefully you can figure something out.

Thats another interesting problem that im trying solving for as well. I made an attempt here Locate anything models do seem very promising!

hey its like CUA without Any closed source provider ?

Its a bit different. Sort of a adapter over CUA

yo, check dms

This is cool! Can the cloud models reuse these recipes to make computer use faster for new actions ?

Yes! They can read the recipes and construct their Spec better. Thats why the spec is useful. It standardises things in a way

Is every action cached locally only? You should find a way to cache it globally so it gets better for everyone the more it’s used. The only thing is beware of data privacy.. perhaps don’t store the data, and only the action or something? Like in your example: {"op":"act","do":"type","text":"{query}"} You could store this exact literal, with query being generic and no user data I really like this though! I’m gonna follow to keep up with it!

Thats a good point! Yeah I think there are small language models which can be used to redact personal information (PII) models But anyways we just need the template only in the axstream format with things being generic Thanks for following lets see where it goes

i'm excited to see where it goes my friend!

Haha So am I!

alas something interesting i saw today

I wish to keep you entertained

This is similar to what stagehand does right?

I think so but this is for computer use and stagehand is I believe for browsers

Yeah exactly. That's what I meant. You're doing stagehand for desktop. Super cool!

Does it work

Clever We're testing this w/ a small router model fine tuned for a corpus of tasks that can route to the correct tool call/workflow But yours makes a lot more sense practically as one may not know all cases upfront so this grows over time

we should benchmark both!

Fabulous Idea!! You should pitch this to frontier labs!!

Lets see if I can reach out to them here on X :D

Say it once slow, say it again instant, that's smart caching ⚡

It is smart yes.

Caching repeated actions like that is such an obvious win once someone actually builds it.

mlx?

love the idea of agents finally learning and remembering locally! so smart. ✨

Moving the repeatable steps to a 350 M local model turns the first call into a slow warm up and the rest into instant actions. The real pain point shows up when the underlying UI changes or lacks an accessibility tree. A lightweight diff layer can keep the cache valid without relearning each time

This is incredible, now I don't have to use codex or warp for STT. I will install this!

It’s still WIP

you're saying it turns tasks into scripts/skill then next time it checks if a script/skill already exists?

This is cool

Maybe

Hey Melinda, congrats for your finds. Amazing and smart idea! Just a question: OpenMausBot has “teach me” feature like grok bot or just compute use? Appreciate for your ideas e open source grok bot alternative.