Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Microsoft released a groundbreaking model that can be used for web automation, with MIT license 🔥👏 OmniParser is a state-of-the-art UI parsing/understanding model that outperforms GPT4V in parsing. 👏

mervenoyann's profile picture

merve

87,305 subscribers

473,069 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

9 Kommentare

Profilbild von merve
mervevor 1 Jahr

Model: Interesting highlight for me was Mind2Web (a benchmark for web navigation) capabilities of the model, which unlocks agentic behavior for RPA agents. no need for hefty web automation pipelines that get broken when the website/app design changes! Amazing work.

Profilbild von merve
mervevor 1 Jahr

Lastly, the authors also fine-tune this model on open-set detection for interactable regions and see if they can use it as a plug-in for VLMs and it actually outperforms off-the-shelf open-set detectors like GroundingDINO. 👏

Profilbild von merve
mervevor 1 Jahr

Here's a bunch of i/o examples for the model ⇓

Profilbild von Sar
Sarvor 1 Jahr

I saw your post and made me think of which I had just come across. Would be interested in hearing from @skyvernai about the possibility of using OmniParser to replace their current approach

Profilbild von Lisan al Gaib
Lisan al Gaibvor 1 Jahr

Incorrect, it has AGPL- 3.0 license since it is based on YOLOv8 by Ultralytics which has AGPL- 3.0 license. You can use it comercially, however your code must be publicly availabe, which makes it comercially unviable again.

Profilbild von Tarek Ayed
Tarek Ayedvor 1 Jahr

It's weird to compare to GPT-4V which is notoriously bad at image understanding and OCR, right? I'd be curious to know how it fares agains 4o, Sonnet, Gemini Flash and Pro, etc.

Profilbild von Johannes
Johannesvor 1 Jahr

Might be nice to combine with anthropic's computer use

Profilbild von Thread Reader App
Thread Reader Appvor 1 Jahr

Your thread is going viral! #TopUnroll 🙏🏼@dl4senses for 🥇unroll

Profilbild von Didier Lacroix
Didier Lacroixvor 1 Jahr

@threadreaderapp unroll

Ähnliche Videos

Microsoft silently updated OmniParser on the hub 👀 60% faster than v1 - sub-second latency on a 4090! "OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent." Bonus: you can try it out for free!
0:46

Microsoft silently updated OmniParser on the hub 👀 60% faster than v1 - sub-second latency on a 4090! "OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent." Bonus: you can try it out for free!

Vaibhav (VB) Srivastav

95,750 Aufrufe • vor 1 Jahr

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇
1:17

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

Xenova

64,553 Aufrufe • vor 1 Jahr

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.
0:25

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

CherryAI Robotics

90,786 Aufrufe • vor 9 Monaten

Everyone is sleeping on this new OCR model! dots-ocr is a new 1.7B vision-language model that achieves SOTA performance on multilingual document parsing. - Supports 100+ languages - Works with both images and PDFs - Handles text, tables, formulas seamlessly 100% open-source.
0:33

Everyone is sleeping on this new OCR model! dots-ocr is a new 1.7B vision-language model that achieves SOTA performance on multilingual document parsing. - Supports 100+ languages - Works with both images and PDFs - Handles text, tables, formulas seamlessly 100% open-source.

Akshay 🚀

251,968 Aufrufe • vor 10 Monaten

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.
0:51

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Google AI

492,732 Aufrufe • vor 7 Monaten

Rekha Gupta ji proves how a Chief Minister can transform a state given to her in Garbage consition, into a clean, thriving model of governance. Amidst opposing forces. 👏👏
1:15

Rekha Gupta ji proves how a Chief Minister can transform a state given to her in Garbage consition, into a clean, thriving model of governance. Amidst opposing forces. 👏👏

Stringg

77,783 Aufrufe • vor 4 Monaten

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: Paper: Playground:
0:55

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: Paper: Playground:

Yichuan Wang

84,334 Aufrufe • vor 21 Tagen

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.
1:13

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,728 Aufrufe • vor 7 Monaten

Introducing DINOv3: a state-of-the-art computer vision model trained with self-supervised learning (SSL) that produces powerful, high-resolution image features. For the first time, a single frozen vision backbone outperforms specialized solutions on multiple long-standing dense prediction tasks. Learn more about DINOv3 here:
1:25

Introducing DINOv3: a state-of-the-art computer vision model trained with self-supervised learning (SSL) that produces powerful, high-resolution image features. For the first time, a single frozen vision backbone outperforms specialized solutions on multiple long-standing dense prediction tasks. Learn more about DINOv3 here:

AI at Meta

899,777 Aufrufe • vor 10 Monaten

👏PUPPETEER👏YOUR👏MODEL!!!👏 It’ll make a world of a difference! ✨
0:36

👏PUPPETEER👏YOUR👏MODEL!!!👏 It’ll make a world of a difference! ✨

NyomiLeigh ✨

300,179 Aufrufe • vor 1 Jahr

Microsoft released Trellis! Trellis is a new 3D model that generates high-quality 3D assets in formats like Radiance Fields, 3D Gaussians, and meshes.
0:36

Microsoft released Trellis! Trellis is a new 3D model that generates high-quality 3D assets in formats like Radiance Fields, 3D Gaussians, and meshes.

Dreaming Tulpa 🥓👑

85,115 Aufrufe • vor 1 Jahr

Today we are announcing Avalon – the best transcription model in the world for prompting AIs. It outperforms the previous state-of-the-art models like ElevenLabs' Scribe and OpenAI's Whisper.
1:38

Today we are announcing Avalon – the best transcription model in the world for prompting AIs. It outperforms the previous state-of-the-art models like ElevenLabs' Scribe and OpenAI's Whisper.

Aqua Voice

472,132 Aufrufe • vor 10 Monaten

This is @OpenCode—an open source Claude Code alternative that: * has a built-in web UI * can use any LLM model, including free ones! In this demo, I'm running OpenCode inside a Daytona cloud sandbox.
0:29

This is @OpenCode—an open source Claude Code alternative that: * has a built-in web UI * can use any LLM model, including free ones! In this demo, I'm running OpenCode inside a Daytona cloud sandbox.

James Murdza

69,669 Aufrufe • vor 5 Monaten

GUJARAT MODEL EXPOSED 🚨 Stampede-like situation in Bharuch, Gujarat where thousands of youth gathered for a private job. Imagine the state of unemployment in PM Modi’s home state 👏
0:22

GUJARAT MODEL EXPOSED 🚨 Stampede-like situation in Bharuch, Gujarat where thousands of youth gathered for a private job. Imagine the state of unemployment in PM Modi’s home state 👏

Ankit Mayank

216,822 Aufrufe • vor 2 Jahren

‼️🎈🔥 Hats off to Nomsa Masuku 👏👏👏👏🙌🙌🙌👏👏👏🔥 She is the kind of woman who should be leading IPID‼️ Nomsa left the Commissioners with a lump in their throats, so overwhelmed that they were momentarily lost for words. 🙌💥 #MadlangaCommission
5:59

‼️🎈🔥 Hats off to Nomsa Masuku 👏👏👏👏🙌🙌🙌👏👏👏🔥 She is the kind of woman who should be leading IPID‼️ Nomsa left the Commissioners with a lump in their throats, so overwhelmed that they were momentarily lost for words. 🙌💥 #MadlangaCommission

Siyakhula

31,138 Aufrufe • vor 4 Monaten

"This whole story and narrative that Google has already won is going to be harder to tell when the best model is actually an OpenAI model," D.A. Davidson's Gil Luria says. He also says that OpenAI will launch that "state-of-the-art" model "over the next few days, weeks."
2:38

"This whole story and narrative that Google has already won is going to be harder to tell when the best model is actually an OpenAI model," D.A. Davidson's Gil Luria says. He also says that OpenAI will launch that "state-of-the-art" model "over the next few days, weeks."

Yahoo Finance

12,410 Aufrufe • vor 4 Monaten

Big News! Meta just released Segment Anything, a new AI model that can "cut out" any object, in any image/video, with a single click. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks.
0:22

Big News! Meta just released Segment Anything, a new AI model that can "cut out" any object, in any image/video, with a single click. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks.

Lior Alexander

290,190 Aufrufe • vor 3 Jahren

Ok, Kokoro is insane. 🤯 This AI is a groundbreaking TTS model with just 82M parameters. It outperforms larger models and generates minutes of speech in seconds. Oh, and it’s open source! Try it and see for yourself: 👇
0:24

Ok, Kokoro is insane. 🤯 This AI is a groundbreaking TTS model with just 82M parameters. It outperforms larger models and generates minutes of speech in seconds. Oh, and it’s open source! Try it and see for yourself: 👇

Min Choi

418,016 Aufrufe • vor 1 Jahr

🚀UniWorld: a unified model that skips VAEs and uses semantic features from SigLIP! Using just 1% of BAGEL’s data, it outperforms on image editing and excels in understanding & generation. 🌟Now data, model, training & evaluation script are open-source!
2:06

🚀UniWorld: a unified model that skips VAEs and uses semantic features from SigLIP! Using just 1% of BAGEL’s data, it outperforms on image editing and excels in understanding & generation. 🌟Now data, model, training & evaluation script are open-source!

Bin Lin

22,327 Aufrufe • vor 1 Jahr

This is Gemini 3: our most intelligent model that helps you learn, build and plan anything. It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵
1:56

This is Gemini 3: our most intelligent model that helps you learn, build and plan anything. It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵

Google DeepMind

1,692,531 Aufrufe • vor 7 Monaten