..jesus open interpreter's first vision model, piloting my 8gb M1 macbook. 100% offline.
@hellokillian32K followers
372,183 views
10 Comments
Comments from the original post on X (Twitter)

@mayfer
are there any plans for hierarchical vision models? i.e. if there are multiple play buttons on the screen, and i say "click play on youtube", it knows to first isolate youtube window for the next inference on the vision model?
30 likes

@hellokillian
actually yes, this exactly! soon, OI will be focused on the active window + let the LLM programmatically switch active windows. have been experimenting with the right way to do this. dramatically improves + speeds up vision model inference too, way less pixels.
50 likes

@GuliMoreno
This is so cool! Is this something like the Large Action Model proposed by @rabbit_hmi, where in a future state you could have it interact in the background with any app?
12 likes

@hellokillian
@rabbit_hmi thanks guli! yes, exactly what we're building—the 01 is an open-source Rabbit R1, this model is the fruit of that project.
46 likes

@josh_bickett
Nice, y'all got icons working
8 likes

@hellokillian
yes! also you are a legend josh. would love to play with this model in the self-operating computer repo, building on the incredible advances you've made there. next OI should expose the model pretty cleanly, something like interpreter.point(screenshot_base64, query) -> coords
17 likes

@romechenko
This is brilliant. I can see a integration testing framework built on top of this
7 likes

@hellokillian
yes!! great stuff in a similar vein at powered by OI
19 likes

@123wimi
You're the best killian
6 likes

@hellokillian
thanks happy!
3 likes