Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Microsoft silently updated OmniParser on the hub 👀 60% faster than v1 - sub-second latency on a 4090! "OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent." Bonus: you can try it out for free!

Vaibhav (VB) Srivastav

35,855 subscribers

95,750 views • 1 year ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

11 Comments

Vaibhav (VB) Srivastav1 year ago

Model checkpoint:

Vaibhav (VB) Srivastav1 year ago

Try out the model for free here:

AssemblyAI1 year ago

Our speech-to-text models are the most accurate on the market with top rankings across industry benchmarks. - The highest accuracy rates—up to 95% - Up to 30% fewer hallucinations than other leaders - Low latency—63 minutes converts in 35 seconds Try via API for free today 👇

Rohith1 year ago

Super hard to use omnitool and its very slow even on my 4080. Like taking 1min+ just for the Image parsing.

Vaibhav (VB) Srivastav1 year ago

are you using the GPU, could be that you're running the op via CPU

moonbi 🇪🇺🪢1 year ago

The problem is u need huge gpu till run it

Cybersphere AI1 year ago

Setup is a nightmare on Windows. The setup is: FUSE-based (horribly slow) Assumes Linux KVM is present (Not true on Windows) Not built with Windows users in mind (requiring modifications to scripts) The devs need to release a setup procedure optimised for Windows.

Shabah1 year ago

This is really interesting

Electe1 year ago

@huggingface @huggingface, incredible improvements with OmniParser. Speed like this can truly enhance workflows and drive productivity. 🚀 #Innovation

Tenkaizen1 year ago

That's impressive progress. Faster parsing opens up so many possibilities for UI agents

Korey 🧘🏾‍♂️1 year ago

such a project, want to build some code samples with this.

Related Videos

Microsoft released a groundbreaking model that can be used for web automation, with MIT license 🔥👏 OmniParser is a state-of-the-art UI parsing/understanding model that outperforms GPT4V in parsing. 👏

Microsoft released a groundbreaking model that can be used for web automation, with MIT license 🔥👏 OmniParser is a state-of-the-art UI parsing/understanding model that outperforms GPT4V in parsing. 👏

merve

473,157 views • 1 year ago

Microsoft just released an impressive tool OmniParser V2 can turn any LLM into an agent capable of using a computer 🔥 You can enable GPT-4o, DeepSeek R1, Sonnet 3.5, Qwen... to understand what's on your screen and take actions. 100% free & open source

Microsoft just released an impressive tool OmniParser V2 can turn any LLM into an agent capable of using a computer 🔥 You can enable GPT-4o, DeepSeek R1, Sonnet 3.5, Qwen... to understand what's on your screen and take actions. 100% free & open source

Paul Couvert

459,967 views • 1 year ago

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

Xenova

64,553 views • 1 year ago

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

73,026 views • 2 months ago

AG-UI makes building agentic applications dramatically easier. Here's how it works. This is a model for a simple chatbot: User → LLM → Response But interactive agents that render UI, pause for approvals, and ask users for input need a much more complex model. When building these agents, a response from the LLM will include a series of state changes as the agent runs: • Agent started a task • Agent called a tool • Agent updated its state • Agent streams these tokens • Agent is waiting on a human • Agent is resuming the task The Agent-User Interaction Protocol (AG-UI) treats the LLM response as a stream of events rather than a text endpoint. In practice, here is what you get as an agent runs: 1. Lifecycle events so your UI knows where the agent is. 2. Text messages that stream tokens. 3. Tool calls so your UI can prefill a form with any required arguments. 4. State updates that keep your UI in sync with the agent. 5. Special events for human approvals, rich media, and custom needs. All of these events travel over standard transports (SSE, WebSockets, or plain HTTP) as JSON. As a result, you can build a frontend that stays in sync with the agent's progress without having to invent a custom process to make this happen. For example, building a human-in-the-loop workflow becomes an off-the-shelf component you can integrate rather than build from scratch. CopilotKit🪁 is the creator of AG-UI, and you can use it when building frontend applications pretty much anywhere: • React • Angular • Vue • React Native • Slack • Teams • Discord • WhatsApp • Telegram Here is the link for you to check it out: Thanks to the CopilotKit team for partnering with me on this post.

AG-UI makes building agentic applications dramatically easier. Here's how it works. This is a model for a simple chatbot: User → LLM → Response But interactive agents that render UI, pause for approvals, and ask users for input need a much more complex model. When building these agents, a response from the LLM will include a series of state changes as the agent runs: • Agent started a task • Agent called a tool • Agent updated its state • Agent streams these tokens • Agent is waiting on a human • Agent is resuming the task The Agent-User Interaction Protocol (AG-UI) treats the LLM response as a stream of events rather than a text endpoint. In practice, here is what you get as an agent runs: 1. Lifecycle events so your UI knows where the agent is. 2. Text messages that stream tokens. 3. Tool calls so your UI can prefill a form with any required arguments. 4. State updates that keep your UI in sync with the agent. 5. Special events for human approvals, rich media, and custom needs. All of these events travel over standard transports (SSE, WebSockets, or plain HTTP) as JSON. As a result, you can build a frontend that stays in sync with the agent's progress without having to invent a custom process to make this happen. For example, building a human-in-the-loop workflow becomes an off-the-shelf component you can integrate rather than build from scratch. CopilotKit🪁 is the creator of AG-UI, and you can use it when building frontend applications pretty much anywhere: • React • Angular • Vue • React Native • Slack • Teams • Discord • WhatsApp • Telegram Here is the link for you to check it out: Thanks to the CopilotKit team for partnering with me on this post.

Santiago

17,438 views • 11 days ago

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,013 views • 1 month ago

Introducing the Generative UI Research Canvas ✨ Every agent needs a powerful UI Combine LangChain, Tavily, Tako and CopilotKit🪁 and you get a LangChain-based research agent with a Generative UI-driven frontend. When agents can explore the underlying data, research becomes more trustworthy and useful. How does this work? - LangChain manages the agent workflow and tool orchestration. - Tako converts retrieved data into structured, visual components - Tavily provides agentic search - CopilotKit renders those components in the UI and keeps the user in the loop while the agent is working So instead of a boring and static report, the output is an interactive research canvas that evolves as the agent runs. 2026 is the year where users want to see the output, not read it... Generative UI makes this possible for agentic applications. Tutorial: Open-Source Repo: Docs:

Introducing the Generative UI Research Canvas ✨ Every agent needs a powerful UI Combine LangChain, Tavily, Tako and CopilotKit🪁 and you get a LangChain-based research agent with a Generative UI-driven frontend. When agents can explore the underlying data, research becomes more trustworthy and useful. How does this work? - LangChain manages the agent workflow and tool orchestration. - Tako converts retrieved data into structured, visual components - Tavily provides agentic search - CopilotKit renders those components in the UI and keeps the user in the loop while the agent is working So instead of a boring and static report, the output is an interactive research canvas that evolves as the agent runs. 2026 is the year where users want to see the output, not read it... Generative UI makes this possible for agentic applications. Tutorial: Open-Source Repo: Docs:

Atai Barkai

11,997 views • 5 months ago

Microsoft Teams just got Generative UI But this is just the tip of the iceberg. AG-UI is the protocol connecting any agent to any user on any app, topped with Generative UI. That means your agent can render: - forms - buttons - charts - approval flows Right inside the chat, on the fly. But it's not limited to in-app copilots. What's special is that you can work with any agent using the same system and bring them into any surface, including Slack, WhatsApp, Discord and more. This is exactly why Hermes and OpenClaw took off. As users of these agents, we don't like walls of text, and it's a pain to get when you can have way better UX. CopilotKit🪁 + AG-UI handle that interaction layer. If you want to try out Agentic UI in Microsoft Teams, you can get early access here:

Microsoft Teams just got Generative UI But this is just the tip of the iceberg. AG-UI is the protocol connecting any agent to any user on any app, topped with Generative UI. That means your agent can render: - forms - buttons - charts - approval flows Right inside the chat, on the fly. But it's not limited to in-app copilots. What's special is that you can work with any agent using the same system and bring them into any surface, including Slack, WhatsApp, Discord and more. This is exactly why Hermes and OpenClaw took off. As users of these agents, we don't like walls of text, and it's a pain to get when you can have way better UX. CopilotKit🪁 + AG-UI handle that interaction layer. If you want to try out Agentic UI in Microsoft Teams, you can get early access here:

Atai Barkai

23,776 views • 25 days ago

Introducing Shadify: Generative UI built on ShadCN Simply describe a UI and watch your LangChain agent compose from @ShadCN on the fly, using AG-UI and CopilotKit🪁. Then export it as React code. Repo below. Try it out here:

Introducing Shadify: Generative UI built on ShadCN Simply describe a UI and watch your LangChain agent compose from @ShadCN on the fly, using AG-UI and CopilotKit🪁. Then export it as React code. Repo below. Try it out here:

Atai Barkai

138,046 views • 3 months ago

From screenshot to functional UI in no time. Powered by Gemini 3 Flash, Antigravity can reproduce mockups with high fidelity. Just drop a screenshot, ask to rebuild as a Flutter UI, and watch it launch on your device.

From screenshot to functional UI in no time. Powered by Gemini 3 Flash, Antigravity can reproduce mockups with high fidelity. Just drop a screenshot, ask to rebuild as a Flutter UI, and watch it launch on your device.

Google Antigravity

394,843 views • 5 months ago

How does Netflix know when you try to take a screenshot? Turns out it doesn’t. Rather than a simple piece of software recognizing a screenshot attempt and blacking out the screen, what’s going on is actually happening at the hardware level.

How does Netflix know when you try to take a screenshot? Turns out it doesn’t. Rather than a simple piece of software recognizing a screenshot attempt and blacking out the screen, what’s going on is actually happening at the hardware level.

tech.explain

487,149 views • 2 months ago

Let's give this another go... 👀 Latest comparison of submenu performance for Base UI vs. bazza/ui <𝙼𝚎𝚗𝚞>. Updated to v1.2.0 — I was told v1.0.0 contained a temporary regression of this exact functionality. Base UI feels faster than before ⚡ Let me know what you think.

Let's give this another go... 👀 Latest comparison of submenu performance for Base UI vs. bazza/ui <𝙼𝚎𝚗𝚞>. Updated to v1.2.0 — I was told v1.0.0 contained a temporary regression of this exact functionality. Base UI feels faster than before ⚡ Let me know what you think.

kian bazza

39,105 views • 4 months ago

We are witnessing a fundamental shift in how software is built. Over the decades, UI was static and predictable. But AI is changing the fundamental nature of the interface. We are moving away from fixed UI and toward a new standard: Generative UI. Generative UI lets agents participate in the UI, not just the conversation. It’s a dynamic participant in the user experience. Depending on the complexity of your app, Generative UI generally falls into three categories: 1. Static (AG-UI): UI is chosen from a fixed set of hand-built components 2. Open-Ended (MCP Apps): agent sends arbitrary content like raw HTML or iframes for total flexibility. 3. Declarative (A2UI): agent and frontend communicate via a structured UI spec for dynamic widgets and forms. The AG-UI Protocol by CopilotKit is the real breakthrough. AG-UI acts as the universal transport layer. It provides the integrations and middleware for every Gen-UI spec, making it trivial to bring dynamic components into your existing app. CopilotKit provides the go-to ecosystem for building agentic applications and is the React client of AG-UI. It is designed to be deeply integrated, allowing developers to bring their own frameworks or protocols to the table. Play around with different types of Generative UI here → Check out the video attached to see Generative UI in action. Appreciate the CopilotKit team for collaborating with me on this deep dive.

We are witnessing a fundamental shift in how software is built. Over the decades, UI was static and predictable. But AI is changing the fundamental nature of the interface. We are moving away from fixed UI and toward a new standard: Generative UI. Generative UI lets agents participate in the UI, not just the conversation. It’s a dynamic participant in the user experience. Depending on the complexity of your app, Generative UI generally falls into three categories: 1. Static (AG-UI): UI is chosen from a fixed set of hand-built components 2. Open-Ended (MCP Apps): agent sends arbitrary content like raw HTML or iframes for total flexibility. 3. Declarative (A2UI): agent and frontend communicate via a structured UI spec for dynamic widgets and forms. The AG-UI Protocol by CopilotKit is the real breakthrough. AG-UI acts as the universal transport layer. It provides the integrations and middleware for every Gen-UI spec, making it trivial to bring dynamic components into your existing app. CopilotKit provides the go-to ecosystem for building agentic applications and is the React client of AG-UI. It is designed to be deeply integrated, allowing developers to bring their own frameworks or protocols to the table. Play around with different types of Generative UI here → Check out the video attached to see Generative UI in action. Appreciate the CopilotKit team for collaborating with me on this deep dive.

Pratham

33,920 views • 5 months ago

I hated spending minutes trying to explain which element I meant in a long prompt, so I shipped something to fix it. Inspector’s new precision UI select tool connects the element on your screen to its exact source code, saving you time on prompting.

I hated spending minutes trying to explain which element I meant in a long prompt, so I shipped something to fix it. Inspector’s new precision UI select tool connects the element on your screen to its exact source code, saving you time on prompting.

mikeBuildsMore

12,343 views • 8 months ago

This Claude 3.7 UI workflow is awesome And many vibe coders not aware of it: 1. Have Claude 3.7 to design whole UI screen by screen 2. Ask it to make a plan, break down into components 3. Turn into actual project By having Claude focus on just UI generation, it does a great job You can build beautiful UI without shadcn If you want to learn more about vibe coding beyond basics, join my community here: (also get free access to 10xCoder.Dev)

This Claude 3.7 UI workflow is awesome And many vibe coders not aware of it: 1. Have Claude 3.7 to design whole UI screen by screen 2. Ask it to make a plan, break down into components 3. Turn into actual project By having Claude focus on just UI generation, it does a great job You can build beautiful UI without shadcn If you want to learn more about vibe coding beyond basics, join my community here: (also get free access to 10xCoder.Dev)

Jason Zhou

745,638 views • 1 year ago

Microsoft 🤝 AG-UI Protocol Microsoft Agent Framework is now AG-UI compatible! 🎉 Connect your MS Agent Framework .NET agents into fullstack applications with Python coming soon 👀 MS Agent Framework takes care of the agentic backend, AG-UI connects your agents to the frontend. Features include: - Chat UI - Generative UI - State synchronization - Human-in-the-loop - Frontend Tool Calls All the building blocks you need to take your agents from the backend and into a fullstack agentic application. Available in React & Angular via CopilotKit, and Rust, Kotlin, Rust, Java and .NET via community SDKs Big shoutout to the Microsoft & CopilotKit teams, who shipped this highly requested integration at breakneck speeds. More coming soon 👀 🙏 Javier Nelson, Joonseok Oh, Daniel Roth, Evan Mattson, Tyler Slaton, Brandon McConnell, Markus Ecker Read more about our collaboration: Get started: Check out the Dojo:

Microsoft 🤝 AG-UI Protocol Microsoft Agent Framework is now AG-UI compatible! 🎉 Connect your MS Agent Framework .NET agents into fullstack applications with Python coming soon 👀 MS Agent Framework takes care of the agentic backend, AG-UI connects your agents to the frontend. Features include: - Chat UI - Generative UI - State synchronization - Human-in-the-loop - Frontend Tool Calls All the building blocks you need to take your agents from the backend and into a fullstack agentic application. Available in React & Angular via CopilotKit, and Rust, Kotlin, Rust, Java and .NET via community SDKs Big shoutout to the Microsoft & CopilotKit teams, who shipped this highly requested integration at breakneck speeds. More coming soon 👀 🙏 Javier Nelson, Joonseok Oh, Daniel Roth, Evan Mattson, Tyler Slaton, Brandon McConnell, Markus Ecker Read more about our collaboration: Get started: Check out the Dojo:

CopilotKit🪁

40,520 views • 8 months ago

🎨 Design freely with Agent 4. - Design while the agent builds. - Explore multiple UI variants on an infinite canvas. - Make precise UI edits that you can apply directly to your app.

🎨 Design freely with Agent 4. - Design while the agent builds. - Explore multiple UI variants on an infinite canvas. - Make precise UI edits that you can apply directly to your app.

Replit ⠕

184,057 views • 3 months ago

Generative UI will be the next big shift in AI. Where your product UI is dynamically generated, based on user intent and context. If you are a developer building AI apps, you need to hear this out:

Generative UI will be the next big shift in AI. Where your product UI is dynamically generated, based on user intent and context. If you are a developer building AI apps, you need to hear this out:

Pratham

119,196 views • 8 months ago

i built the first ever email to webhook platform sub 300ms processing, beautiful UI and open source try it for free below

i built the first ever email to webhook platform sub 300ms processing, beautiful UI and open source try it for free below

ryan vogel

99,642 views • 1 year ago

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Hugo

11,671 views • 2 months ago