Skip navigation
Introducing Tarsier 🙈, an open source Python library to enable web interaction with multi-modal LLMs…
0:00 / 0:33

Introducing Tarsier 🙈, an open source Python library to enable web interaction with multi-modal LLMs…

Rohan Pandey

@khoomeik43K followers

Visit on X
350 likesDownloadOriginal post

75,135 views

Introducing Tarsier 🙈, an open source Python library to enable web interaction with multi-modal LLMs like GPT4! Here’s a demo of a Tarsier agent navigating through google to watch the OpenAI Dev day announcement:

11 Comments

Comments from the original post on X (Twitter)

  • Rohan Pandey's profile picture

    @khoomeik

    Tarsier provides two fundamental utilities: 1. ability to tag interactable elements with a unique id. This allows LLMs to better understand what the elements they can take actions upon are, and this also provides a mapping back from the LLMs choice to the underlying element.

    6 likes

  • Rohan Pandey's profile picture

    @khoomeik

    2. ability to parse a screenshot into a spatially aware OCR text representation. In doing this, we’re able to even use GPT4 *without vision* (or any other text-only LLM) for web tasks.

    9 likes

  • Rohan Pandey's profile picture

    @khoomeik

    To make it as easy as possible to get started, we’ve created a few cookbooks to demonstrate Tarsier usage with popular LLM Libraries like @Langchain and @LlamaIndex. You can access them here and play with Tarsier yourself:

    12 likes

  • Rohan Pandey's profile picture

    @khoomeik

    As mentioned earlier, this is all open source! You can check out the repo/read release notes here:

    23 likes

  • Rainmaker's profile picture

    @RainmakerTrade

    Explore initial jobless claims as a crucial labor market indicator. Learn how to analyze trends using Python and FRED. Understand the signals that drive financial and economic strategies in my latest free Substack post.

    25 likes

  • Graham Neubig's profile picture

    @gneubig

    Hey Rohan, this looks really cool! If you'd be interested in hooking it up to WebArena ( to evaluate and iterate on web nav tasks I'd be happy to chat about how to do this (via email or DM).

    4 likes

  • Rohan Pandey's profile picture

    @khoomeik

    Thanks! Yes we’ve been playing around with WebArena a lot internally and actually built some eval tooling for it too: Will be in touch with Shuyan as needed!

    3 likes

  • Sagar Shah's profile picture

    @sgzsh269

    This looks great! Though one big limitation of solely using the vision approach is that the model won't know about the non-visible elements such as options in a select element and elements that are hidden in an accordion. Enterprise apps can have many instances of those non-visible elements so a more robust approach would still require processing the HTML

    3 likes

  • Rohan Pandey's profile picture

    @khoomeik

    True for a naive OCR approach, but you fine-tune your OCR model on common web logos. GPT-4V on the other hand is usually capable of identifying e.g. accordion menus, and Tarsier will tag them as interactable. If humans can do it w/o HTML, VLMs should be able to too.

    1 like

  • Yohei's profile picture

    @yoheinakajima

    Nice!

    2 likes

  • Rohan Pandey's profile picture

    @khoomeik

    Thanks! Would love to have you try our web data extraction agent as we get closer to launch 🫡

    2 likes