OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

🔗 Demo & source code:

@huggingface Building AI browser automation + scraping on Chrome Extension here But I'm curious, what makes OmniParser different from just reading DOM elements (or accessibility tree) directly and drawing it to an image? That’s how I’m doing it currently

@huggingface Can it be used to infer from the whole desktop image, while running like this inside the browser, or would it be restricted to the browser window?

@huggingface You can use the Screen Capture API

@huggingface Woah 🤯

@huggingface The inference speed for this is not usable in practice tho, could we support that can just get a bounding box with an element description?

@huggingface Another massive w You are on fire transformers Js is looking great!

@huggingface What it can do?

Given a screenshot of a user interface, it detects interactable icons/elements on the page (w/ bounding boxes) and generates meaningful descriptions for each. It's meant to enhance the ability of a multimodal model (like GPT-4V) to generate actions (so it can interact with the page). Links to learn more: - GitHub repo: - Blog post: - Project page:

@huggingface I will do it! Maybe not first but will do it