Introducing Tarsier 🙈, an open source Python library to enable web interaction with multi-modal LLMs like GPT4! Here’s a demo of a Tarsier agent navigating through google to watch the OpenAI Dev day announcement:

Tarsier provides two fundamental utilities: 1. ability to tag interactable elements with a unique id. This allows LLMs to better understand what the elements they can take actions upon are, and this also provides a mapping back from the LLMs choice to the underlying element.

2. ability to parse a screenshot into a spatially aware OCR text representation. In doing this, we’re able to even use GPT4 *without vision* (or any other text-only LLM) for web tasks.

To make it as easy as possible to get started, we’ve created a few cookbooks to demonstrate Tarsier usage with popular LLM Libraries like @Langchain and @LlamaIndex. You can access them here and play with Tarsier yourself:

As mentioned earlier, this is all open source! You can check out the repo/read release notes here:

Explore initial jobless claims as a crucial labor market indicator. Learn how to analyze trends using Python and FRED. Understand the signals that drive financial and economic strategies in my latest free Substack post.

Hey Rohan, this looks really cool! If you'd be interested in hooking it up to WebArena ( to evaluate and iterate on web nav tasks I'd be happy to chat about how to do this (via email or DM).

Thanks! Yes we’ve been playing around with WebArena a lot internally and actually built some eval tooling for it too: Will be in touch with Shuyan as needed!

This looks great! Though one big limitation of solely using the vision approach is that the model won't know about the non-visible elements such as options in a select element and elements that are hidden in an accordion. Enterprise apps can have many instances of those non-visible elements so a more robust approach would still require processing the HTML

True for a naive OCR approach, but you fine-tune your OCR model on common web logos. GPT-4V on the other hand is usually capable of identifying e.g. accordion menus, and Tarsier will tag them as interactable. If humans can do it w/o HTML, VLMs should be able to too.

Nice!

Thanks! Would love to have you try our web data extraction agent as we get closer to launch 🫡