正在加载视频...

视频加载失败

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

64,560 次观看 • 1 年前 •via X (Twitter)

10 条评论

Xenova 的头像
Xenova1 年前

🔗 Demo & source code:

An 的头像
An1 年前

@huggingface Building AI browser automation + scraping on Chrome Extension here But I'm curious, what makes OmniParser different from just reading DOM elements (or accessibility tree) directly and drawing it to an image? That’s how I’m doing it currently

Rod 的头像
Rod1 年前

@huggingface Can it be used to infer from the whole desktop image, while running like this inside the browser, or would it be restricted to the browser window?

Xenova 的头像
Xenova1 年前

@huggingface You can use the Screen Capture API

Caleb 的头像
Caleb1 年前

@huggingface Woah 🤯

jian 的头像
jian1 年前

@huggingface The inference speed for this is not usable in practice tho, could we support that can just get a bounding box with an element description?

lee 的头像
lee1 年前

@huggingface Another massive w You are on fire transformers Js is looking great!

llmstock.com mc 的头像
llmstock.com mc1 年前

@huggingface What it can do?

Xenova 的头像
Xenova1 年前

Given a screenshot of a user interface, it detects interactable icons/elements on the page (w/ bounding boxes) and generates meaningful descriptions for each. It's meant to enhance the ability of a multimodal model (like GPT-4V) to generate actions (so it can interact with the page). Links to learn more: - GitHub repo: - Blog post: - Project page:

justboulatbek 的头像
justboulatbek1 年前

@huggingface I will do it! Maybe not first but will do it

相关视频