Skip navigation
..jesus open interpreter's first vision model, piloting my 8gb M1 macbook. 100% offline.
0:00 / 2:08

..jesus open interpreter's first vision model, piloting my 8gb M1 macbook. 100% offline.

killian

@hellokillian32K followers

Visit on X
2,616 likesDownloadOriginal post

372,183 views

..jesus open interpreter's first vision model, piloting my 8gb M1 macbook. 100% offline. this will be inside every computer in the world.

10 Comments

Comments from the original post on X (Twitter)

  • murat 🍥's profile picture

    @mayfer

    are there any plans for hierarchical vision models? i.e. if there are multiple play buttons on the screen, and i say "click play on youtube", it knows to first isolate youtube window for the next inference on the vision model?

    30 likes

  • killian's profile picture

    @hellokillian

    actually yes, this exactly! soon, OI will be focused on the active window + let the LLM programmatically switch active windows. have been experimenting with the right way to do this. dramatically improves + speeds up vision model inference too, way less pixels.

    50 likes

  • Guli Moreno's profile picture

    @GuliMoreno

    This is so cool! Is this something like the Large Action Model proposed by @rabbit_hmi, where in a future state you could have it interact in the background with any app?

    12 likes

  • killian's profile picture

    @hellokillian

    @rabbit_hmi thanks guli! yes, exactly what we're building—the 01 is an open-source Rabbit R1, this model is the fruit of that project.

    46 likes

  • Josh's profile picture

    @josh_bickett

    Nice, y'all got icons working

    8 likes

  • killian's profile picture

    @hellokillian

    yes! also you are a legend josh. would love to play with this model in the self-operating computer repo, building on the incredible advances you've made there. next OI should expose the model pretty cleanly, something like interpreter.point(screenshot_base64, query) -> coords

    17 likes

  • Roman Pshichenko's profile picture

    @romechenko

    This is brilliant. I can see a integration testing framework built on top of this

    7 likes

  • killian's profile picture

    @hellokillian

    yes!! great stuff in a similar vein at powered by OI

    19 likes

  • Happy's profile picture

    @123wimi

    You're the best killian

    6 likes

  • killian's profile picture

    @hellokillian

    thanks happy!

    3 likes