正在加载视频...

视频加载失败

computer use agents are slow because they think too much they make round trips to an LLM. every single time. even for a command you've said a hundred times before. that's why it always feels slow. my mac now remembers. the first time i say something new, like "search...

14,235 次观看 • 3 个月前 •via X (Twitter)

42 条评论

Milind S 的头像
Milind S3 个月前

this is what im playing around with

🥔🥔🥔 的头像
🥔🥔🥔3 个月前

👀

Milind S 的头像
Milind S3 个月前

:D

joshua 的头像
joshua3 个月前

smells like there’s a big if true idea here, cool stuff

Milind S 的头像
Milind S3 个月前

Finding out in realtime :D

Queue 的头像
Queue3 个月前

the latency problem is less “the model is slow” and more “the model is being asked to rediscover the same procedure.” caching the routine path feels like the right boundary: keep novel decisions in the loop, move deterministic commands out of it.

Milind S 的头像
Milind S3 个月前

It isnt quite transparent of how caching works with LLMs. This gives me more control over things. But yes I think its a similar concept that I’m implementing

Queue 的头像
Queue3 个月前

yeah, cache hits are cute until you cannot see what it actually did.

Queue 的头像
Queue3 个月前

they also dont really disclose cache costs,. atleast to my understanding

PJ 的头像
PJ3 个月前

Good stuff, hope u can crack computer use with local models working with the cua team. Challenge - can you think of how to solve the problem of apps which don't surface accessibility trees such as most enterprise/business software? Or even apps running in VM's / remote desktops. Closest thing I can think of is using a harness which utilizes a thinking model & a model like Locate Anything 3B. But you are the expert here, hopefully you can figure something out.

Milind S 的头像
Milind S3 个月前

Thats another interesting problem that im trying solving for as well. I made an attempt here Locate anything models do seem very promising!

Raj 的头像
Raj3 个月前

hey its like CUA without Any closed source provider ?

Milind S 的头像
Milind S3 个月前

Its a bit different. Sort of a adapter over CUA

rahul (km) 的头像
rahul (km)3 个月前

yo, check dms

A.K 的头像
A.K3 个月前

This is cool! Can the cloud models reuse these recipes to make computer use faster for new actions ?

Milind S 的头像
Milind S3 个月前

Yes! They can read the recipes and construct their Spec better. Thats why the spec is useful. It standardises things in a way

Gary 的头像
Gary3 个月前

Is every action cached locally only? You should find a way to cache it globally so it gets better for everyone the more it’s used. The only thing is beware of data privacy.. perhaps don’t store the data, and only the action or something? Like in your example: {"op":"act","do":"type","text":"{query}"} You could store this exact literal, with query being generic and no user data I really like this though! I’m gonna follow to keep up with it!

Milind S 的头像
Milind S3 个月前

Thats a good point! Yeah I think there are small language models which can be used to redact personal information (PII) models But anyways we just need the template only in the axstream format with things being generic Thanks for following lets see where it goes

Gary 的头像
Gary3 个月前

i'm excited to see where it goes my friend!

Milind S 的头像
Milind S3 个月前

Haha So am I!

Paul Levi 的头像
Paul Levi3 个月前

alas something interesting i saw today

Milind S 的头像
Milind S3 个月前

I wish to keep you entertained

Ben Freed 的头像
Ben Freed3 个月前

This is similar to what stagehand does right?

Milind S 的头像
Milind S3 个月前

I think so but this is for computer use and stagehand is I believe for browsers

Ben Freed 的头像
Ben Freed3 个月前

Yeah exactly. That's what I meant. You're doing stagehand for desktop. Super cool!

ˏˋNordˎˊ 的头像
ˏˋNordˎˊ3 个月前

Does it work

Harlaksh Singh 的头像
Harlaksh Singh3 个月前

Clever We're testing this w/ a small router model fine tuned for a corpus of tasks that can route to the correct tool call/workflow But yours makes a lot more sense practically as one may not know all cases upfront so this grows over time

Milind S 的头像
Milind S3 个月前

we should benchmark both!

Kishore 的头像
Kishore3 个月前

Fabulous Idea!! You should pitch this to frontier labs!!

Milind S 的头像
Milind S3 个月前

Lets see if I can reach out to them here on X :D

AI Mastery Guide 的头像
AI Mastery Guide3 个月前

Say it once slow, say it again instant, that's smart caching ⚡

Milind S 的头像
Milind S3 个月前

It is smart yes.

AI Mastery Guide 的头像
AI Mastery Guide3 个月前

Caching repeated actions like that is such an obvious win once someone actually builds it.

khushal 的头像
khushal3 个月前

mlx?

Channel Profiler 的头像
Channel Profiler3 个月前

love the idea of agents finally learning and remembering locally! so smart. ✨

Moez Zhioua 的头像
Moez Zhioua3 个月前

Moving the repeatable steps to a 350 M local model turns the first call into a slow warm up and the rest into instant actions. The real pain point shows up when the underlying UI changes or lacks an accessibility tree. A lightweight diff layer can keep the cache valid without relearning each time

Srinivas 的头像
Srinivas3 个月前

This is incredible, now I don't have to use codex or warp for STT. I will install this!

Milind S 的头像
Milind S3 个月前

It’s still WIP

echoes2099 的头像
echoes20993 个月前

you're saying it turns tasks into scripts/skill then next time it checks if a script/skill already exists?

hnp 的头像
hnp3 个月前

This is cool

Milind S 的头像
Milind S3 个月前

Maybe

Carlos Guimaraes 的头像
Carlos Guimaraes2 个月前

Hey Melinda, congrats for your finds. Amazing and smart idea! Just a question: OpenMausBot has “teach me” feature like grok bot or just compute use? Appreciate for your ideas e open source grok bot alternative.

相关视频

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

56,638 次观看 • 1 个月前

Fusion Models by robinhood:0xa92768863a55d8a0591709f7f5e594a249d36ea3, explained properly. Why a panel beats one model: Ask three models the same hard question and they disagree. The disagreement is the information, where they contradict each other is where one of them is wrong, and a judge can see it. OpenRouter shipped exactly this in June and their panels beat any single frontier model. Credit to them and their $10B sale to stripe. Why the same panel for everyone is the wrong shape: You don't have one best model. You have a best model per job. The one you trust for code is not the one you trust for writing, and neither is the one you'd pick for a five-line reply. A panel chosen per request, identical for every user, can't know that. A panel arranged around you can. The number that matters isn't price: It's cost per answer that works. Model A costs $4 per million tokens and gets your job right one time in four. Model B costs $10 and gets it right three times in four. Per answer you actually keep, A costs $16 and B costs $13. The cheap model was the expensive one. Everyone comparing price lists is measuring the wrong thing. You never rate anything: Stopping an answer halfway is a rating. Regenerating is a rating. Switching model and asking again is the loudest rating there is. Every one of those already writes a row we can learn from. The router learns which models earn your credits, per job, and tries a cheaper one now and then to check. Fusion only when it pays: Cost mode tries the cheap model first and escalates only if a judge says it failed. Performance mode fans out to your panel and merges. Most simple prompts never fuse. That's why it costs less, not more. Put it together and you have a personal model: Not trained. Assembled. Your profile from a two-minute questionnaire and everything you've imported from Claude or ChatGPT. Your memory, shared across every model so switching never starts from zero. Your skills, workflows you build once and run on any model. Your scorecard, which models earn your credits per job. One per person. Nobody else has yours. Every model is someone else's; the arrangement is you, and it leaves with you. People aren't loyal to models. They're loyal to results. A moat can't be a commodity, but it can be our own creation.

askr

13,917 次观看 • 1 个月前