Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Microsoft released a groundbreaking model that can be used for web automation, with MIT license 🔥👏 OmniParser is a state-of-the-art UI parsing/understanding model that outperforms GPT4V in parsing. 👏

merve

88,650 subscribers

473,157 views • 1 year ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

9 Comments

merve1 year ago

Model: Interesting highlight for me was Mind2Web (a benchmark for web navigation) capabilities of the model, which unlocks agentic behavior for RPA agents. no need for hefty web automation pipelines that get broken when the website/app design changes! Amazing work.

merve1 year ago

Lastly, the authors also fine-tune this model on open-set detection for interactable regions and see if they can use it as a plug-in for VLMs and it actually outperforms off-the-shelf open-set detectors like GroundingDINO. 👏

merve1 year ago

Here's a bunch of i/o examples for the model ⇓

Sar1 year ago

I saw your post and made me think of which I had just come across. Would be interested in hearing from @skyvernai about the possibility of using OmniParser to replace their current approach

Lisan al Gaib1 year ago

Incorrect, it has AGPL- 3.0 license since it is based on YOLOv8 by Ultralytics which has AGPL- 3.0 license. You can use it comercially, however your code must be publicly availabe, which makes it comercially unviable again.

Tarek Ayed1 year ago

It's weird to compare to GPT-4V which is notoriously bad at image understanding and OCR, right? I'd be curious to know how it fares agains 4o, Sonnet, Gemini Flash and Pro, etc.

Johannes1 year ago

Might be nice to combine with anthropic's computer use

Thread Reader App1 year ago

Your thread is going viral! #TopUnroll 🙏🏼@dl4senses for 🥇unroll

Didier Lacroix1 year ago

@threadreaderapp unroll

Related Videos

Microsoft silently updated OmniParser on the hub 👀 60% faster than v1 - sub-second latency on a 4090! "OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent." Bonus: you can try it out for free!

Microsoft silently updated OmniParser on the hub 👀 60% faster than v1 - sub-second latency on a 4090! "OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent." Bonus: you can try it out for free!

Vaibhav (VB) Srivastav

95,750 views • 1 year ago

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

Xenova

64,553 views • 1 year ago

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

1/ The data parsing market for robotics is just emerging, and it’s already valued in the tens of billions. Data parsing to build data models and software for Robotics is a $20B market in 2024 and is expected to reach $150B+ by 2034 Cherry AI Robotics has built a data parsing tool for robotics, which goes live next week, and will effectively make this project one of the pioneers in data parsing for robotics.

CherryAI Robotics

90,786 views • 10 months ago

Everyone is sleeping on this new OCR model! dots-ocr is a new 1.7B vision-language model that achieves SOTA performance on multilingual document parsing. - Supports 100+ languages - Works with both images and PDFs - Handles text, tables, formulas seamlessly 100% open-source.

Everyone is sleeping on this new OCR model! dots-ocr is a new 1.7B vision-language model that achieves SOTA performance on multilingual document parsing. - Supports 100+ languages - Works with both images and PDFs - Handles text, tables, formulas seamlessly 100% open-source.

Akshay 🚀

252,074 views • 10 months ago

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Today we’re taking a big step on the path toward AGI and releasing Gemini 3— our most intelligent model yet. With Gemini 3, you can bring any idea to life. It is state-of-the-art in reasoning, the best model in the world for multimodal understanding, and our best agentic and vibe coding model.

Google AI

492,825 views • 7 months ago

Rekha Gupta ji proves how a Chief Minister can transform a state given to her in Garbage consition, into a clean, thriving model of governance. Amidst opposing forces. 👏👏

Rekha Gupta ji proves how a Chief Minister can transform a state given to her in Garbage consition, into a clean, thriving model of governance. Amidst opposing forces. 👏👏

Stringg

77,783 views • 5 months ago

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: Paper: Playground:

The web was never meant to be flattened into text. Yet most web RAG systems start by parsing HTML --- a complex and lossy process. 🔥 Introducing PixelRAG: the first RAG system that retrieves and reads 30M+ web pages as pixels. Instead of extracting text, PixelRAG retrieves screenshots and lets a VLM read them directly. PixelRAG not only preserves visual information, but also outperforms text-based RAG on text-only QA benchmarks by +18.1%. Why? (1) HTML-to-text conversion often discards layout, structure, tables, and other useful signals. (2) We continued pretraining a VLM on web page screenshots and turned it into a surprisingly strong visual retriever. (3) Recent VLMs are remarkably good at understanding web pages, often with better accuracy and token efficiency than text-only pipelines. Takeaway: HTML parsing may be one of the biggest self-inflicted bottlenecks in web RAG. Demo below 👇 Code: Paper: Playground:

Yichuan Wang

89,045 views • 1 month ago

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

(1/5) Gemini 3, our most intelligent model, is landing in Google Search today – starting with AI Mode. Excited that this is the first time we’re shipping a new Gemini model in Search on day one! 🚀 In Search, Gemini 3 with generative layouts will make it easy to get a rich understanding of anything on your mind. It has state-of-the-art reasoning, deep multimodal understanding and advanced agentic capabilities. That allows the model to shine when you ask it to explain advanced concepts or ideas – it reasons and can code interactive visuals in real-time. It can tackle your toughest questions like advanced science.

Robby Stein

94,877 views • 7 months ago

👏PUPPETEER👏YOUR👏MODEL!!!👏 It’ll make a world of a difference! ✨

👏PUPPETEER👏YOUR👏MODEL!!!👏 It’ll make a world of a difference! ✨

NyomiLeigh ✨

300,179 views • 1 year ago

Introducing DINOv3: a state-of-the-art computer vision model trained with self-supervised learning (SSL) that produces powerful, high-resolution image features. For the first time, a single frozen vision backbone outperforms specialized solutions on multiple long-standing dense prediction tasks. Learn more about DINOv3 here:

Introducing DINOv3: a state-of-the-art computer vision model trained with self-supervised learning (SSL) that produces powerful, high-resolution image features. For the first time, a single frozen vision backbone outperforms specialized solutions on multiple long-standing dense prediction tasks. Learn more about DINOv3 here:

AI at Meta

900,076 views • 11 months ago

Microsoft released Trellis! Trellis is a new 3D model that generates high-quality 3D assets in formats like Radiance Fields, 3D Gaussians, and meshes.

Microsoft released Trellis! Trellis is a new 3D model that generates high-quality 3D assets in formats like Radiance Fields, 3D Gaussians, and meshes.

Dreaming Tulpa 🥓👑

85,115 views • 1 year ago

Today we are announcing Avalon – the best transcription model in the world for prompting AIs. It outperforms the previous state-of-the-art models like ElevenLabs' Scribe and OpenAI's Whisper.

Today we are announcing Avalon – the best transcription model in the world for prompting AIs. It outperforms the previous state-of-the-art models like ElevenLabs' Scribe and OpenAI's Whisper.

Aqua Voice

472,244 views • 10 months ago

GUJARAT MODEL EXPOSED 🚨 Stampede-like situation in Bharuch, Gujarat where thousands of youth gathered for a private job. Imagine the state of unemployment in PM Modi’s home state 👏

GUJARAT MODEL EXPOSED 🚨 Stampede-like situation in Bharuch, Gujarat where thousands of youth gathered for a private job. Imagine the state of unemployment in PM Modi’s home state 👏

Ankit Mayank

216,822 views • 2 years ago

This is @OpenCode—an open source Claude Code alternative that: * has a built-in web UI * can use any LLM model, including free ones! In this demo, I'm running OpenCode inside a Daytona cloud sandbox.

This is @OpenCode—an open source Claude Code alternative that: * has a built-in web UI * can use any LLM model, including free ones! In this demo, I'm running OpenCode inside a Daytona cloud sandbox.

James Murdza

69,669 views • 6 months ago

‼️🎈🔥 Hats off to Nomsa Masuku 👏👏👏👏🙌🙌🙌👏👏👏🔥 She is the kind of woman who should be leading IPID‼️ Nomsa left the Commissioners with a lump in their throats, so overwhelmed that they were momentarily lost for words. 🙌💥 #MadlangaCommission

‼️🎈🔥 Hats off to Nomsa Masuku 👏👏👏👏🙌🙌🙌👏👏👏🔥 She is the kind of woman who should be leading IPID‼️ Nomsa left the Commissioners with a lump in their throats, so overwhelmed that they were momentarily lost for words. 🙌💥 #MadlangaCommission

Siyakhula

31,138 views • 4 months ago

"This whole story and narrative that Google has already won is going to be harder to tell when the best model is actually an OpenAI model," D.A. Davidson's Gil Luria says. He also says that OpenAI will launch that "state-of-the-art" model "over the next few days, weeks."

"This whole story and narrative that Google has already won is going to be harder to tell when the best model is actually an OpenAI model," D.A. Davidson's Gil Luria says. He also says that OpenAI will launch that "state-of-the-art" model "over the next few days, weeks."

Yahoo Finance

12,410 views • 5 months ago

Big News! Meta just released Segment Anything, a new AI model that can "cut out" any object, in any image/video, with a single click. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks.

Big News! Meta just released Segment Anything, a new AI model that can "cut out" any object, in any image/video, with a single click. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks.

Lior Alexander

290,190 views • 3 years ago

Ok, Kokoro is insane. 🤯 This AI is a groundbreaking TTS model with just 82M parameters. It outperforms larger models and generates minutes of speech in seconds. Oh, and it’s open source! Try it and see for yourself: 👇

Ok, Kokoro is insane. 🤯 This AI is a groundbreaking TTS model with just 82M parameters. It outperforms larger models and generates minutes of speech in seconds. Oh, and it’s open source! Try it and see for yourself: 👇

Min Choi

418,183 views • 1 year ago

The entire AI industry is racing to build the smartest model. Satya Nadella just admitted that is not where the money is. The model is not the product. The harness is. That is the exact line. And it changes what Microsoft is actually competing on. OpenAI, Anthropic, Google, xAI, Meta every frontier lab is pouring hundreds of billions into training compute, chasing the next capability jump. Each betting that raw model intelligence is the moat. Microsoft is doing the opposite. It is building the harness the orchestration layer that sits above the model, connecting it to tools, data, permissions, sub-agents, and enterprise workflows. And it is letting OpenAI, Anthropic, and MAI compete to plug into it. "You need the model. But the model is not the product. The harness is." So do the math on what a harness actually does. A raw model dropped into an enterprise answers questions. That is a chatbot. A harness turns that same model into an agent that reads the SharePoint, edits the ERP entry, pulls the GitHub PR, updates Salesforce, and files the Excel report with the right permissions, the right audit trail, and the right sub-agent for each sub-task. The model provides the intelligence. The harness converts intelligence into work. Now here's where it gets interesting. "Even the best model in the world will feel broken without a great harness. And an okay model with a great harness can feel like magic." If that is true, the enterprise buyer is not buying model quality. The enterprise buyer is buying the harness. Which means model quality becomes a commodity input over time, and harness quality becomes the sustainable moat. Compare that to the strategy the entire frontier lab industry is executing. Everyone else is chasing the numerator raw intelligence. Almost nobody at scale is racing to build the denominator the orchestration layer that determines whether that intelligence can actually be deployed profitably inside a real company. The frontier model race has a 10 to 20 percent chance of producing a single dominant winner. Nadella just told the industry he does not need to be that winner. If OpenAI wins, Microsoft wins. If Anthropic wins, Microsoft wins. If MAI wins, Microsoft wins. If someone Microsoft has never heard of trains a better model in 2027, Microsoft still wins. Because the compute they train on, the harness they get plugged into, the enterprise contracts they get delivered through, and the products they sit inside are all Microsoft. He is not building the best AI model. He is building the layer that the best AI model has to run on to make anyone money. I wonder which position looks more valuable in ten years.

The entire AI industry is racing to build the smartest model. Satya Nadella just admitted that is not where the money is. The model is not the product. The harness is. That is the exact line. And it changes what Microsoft is actually competing on. OpenAI, Anthropic, Google, xAI, Meta every frontier lab is pouring hundreds of billions into training compute, chasing the next capability jump. Each betting that raw model intelligence is the moat. Microsoft is doing the opposite. It is building the harness the orchestration layer that sits above the model, connecting it to tools, data, permissions, sub-agents, and enterprise workflows. And it is letting OpenAI, Anthropic, and MAI compete to plug into it. "You need the model. But the model is not the product. The harness is." So do the math on what a harness actually does. A raw model dropped into an enterprise answers questions. That is a chatbot. A harness turns that same model into an agent that reads the SharePoint, edits the ERP entry, pulls the GitHub PR, updates Salesforce, and files the Excel report with the right permissions, the right audit trail, and the right sub-agent for each sub-task. The model provides the intelligence. The harness converts intelligence into work. Now here's where it gets interesting. "Even the best model in the world will feel broken without a great harness. And an okay model with a great harness can feel like magic." If that is true, the enterprise buyer is not buying model quality. The enterprise buyer is buying the harness. Which means model quality becomes a commodity input over time, and harness quality becomes the sustainable moat. Compare that to the strategy the entire frontier lab industry is executing. Everyone else is chasing the numerator raw intelligence. Almost nobody at scale is racing to build the denominator the orchestration layer that determines whether that intelligence can actually be deployed profitably inside a real company. The frontier model race has a 10 to 20 percent chance of producing a single dominant winner. Nadella just told the industry he does not need to be that winner. If OpenAI wins, Microsoft wins. If Anthropic wins, Microsoft wins. If MAI wins, Microsoft wins. If someone Microsoft has never heard of trains a better model in 2027, Microsoft still wins. Because the compute they train on, the harness they get plugged into, the enterprise contracts they get delivered through, and the products they sit inside are all Microsoft. He is not building the best AI model. He is building the layer that the best AI model has to run on to make anyone money. I wonder which position looks more valuable in ten years.

Vikram M

21,463 views • 6 days ago

🚀UniWorld: a unified model that skips VAEs and uses semantic features from SigLIP! Using just 1% of BAGEL’s data, it outperforms on image editing and excels in understanding & generation. 🌟Now data, model, training & evaluation script are open-source!

🚀UniWorld: a unified model that skips VAEs and uses semantic features from SigLIP! Using just 1% of BAGEL’s data, it outperforms on image editing and excels in understanding & generation. 🌟Now data, model, training & evaluation script are open-source!

Bin Lin

22,327 views • 1 year ago