Introducing Tarsier 🙈, an open source Python library to enable web interaction with multi-modal LLMs…
@khoomeik43K followers
75,135 views
11 Comments
Comments from the original post on X (Twitter)

@khoomeik
Tarsier provides two fundamental utilities: 1. ability to tag interactable elements with a unique id. This allows LLMs to better understand what the elements they can take actions upon are, and this also provides a mapping back from the LLMs choice to the underlying element.
6 likes

@khoomeik
2. ability to parse a screenshot into a spatially aware OCR text representation. In doing this, we’re able to even use GPT4 *without vision* (or any other text-only LLM) for web tasks.
9 likes

@khoomeik
To make it as easy as possible to get started, we’ve created a few cookbooks to demonstrate Tarsier usage with popular LLM Libraries like @Langchain and @LlamaIndex. You can access them here and play with Tarsier yourself:
12 likes

@khoomeik
As mentioned earlier, this is all open source! You can check out the repo/read release notes here:
23 likes

@RainmakerTrade
Explore initial jobless claims as a crucial labor market indicator. Learn how to analyze trends using Python and FRED. Understand the signals that drive financial and economic strategies in my latest free Substack post.
25 likes

@gneubig
Hey Rohan, this looks really cool! If you'd be interested in hooking it up to WebArena ( to evaluate and iterate on web nav tasks I'd be happy to chat about how to do this (via email or DM).
4 likes

@khoomeik
Thanks! Yes we’ve been playing around with WebArena a lot internally and actually built some eval tooling for it too: Will be in touch with Shuyan as needed!
3 likes

@sgzsh269
This looks great! Though one big limitation of solely using the vision approach is that the model won't know about the non-visible elements such as options in a select element and elements that are hidden in an accordion. Enterprise apps can have many instances of those non-visible elements so a more robust approach would still require processing the HTML
3 likes

@khoomeik
True for a naive OCR approach, but you fine-tune your OCR model on common web logos. GPT-4V on the other hand is usually capable of identifying e.g. accordion menus, and Tarsier will tag them as interactable. If humans can do it w/o HTML, VLMs should be able to too.
1 like

@yoheinakajima
Nice!
2 likes

@khoomeik
Thanks! Would love to have you try our web data extraction agent as we get closer to launch 🫡
2 likes
