正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Microsoft silently updated OmniParser on the hub 👀 60% faster than v1 - sub-second latency on a 4090! "OmniParser is a general screen parsing tool, which interprets/converts UI screenshot to structured format, to improve existing LLM based UI agent." Bonus: you can try it out for free!

Vaibhav (VB) Srivastav

35,855 subscribers

95,750 次观看 • 1 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

11 条评论

Vaibhav (VB) Srivastav 的头像

Vaibhav (VB) Srivastav1 年前

Model checkpoint:

Vaibhav (VB) Srivastav 的头像

Vaibhav (VB) Srivastav1 年前

Try out the model for free here:

AssemblyAI 的头像

AssemblyAI1 年前

Our speech-to-text models are the most accurate on the market with top rankings across industry benchmarks. - The highest accuracy rates—up to 95% - Up to 30% fewer hallucinations than other leaders - Low latency—63 minutes converts in 35 seconds Try via API for free today 👇

Rohith 的头像

Rohith1 年前

Super hard to use omnitool and its very slow even on my 4080. Like taking 1min+ just for the Image parsing.

Vaibhav (VB) Srivastav 的头像

Vaibhav (VB) Srivastav1 年前

are you using the GPU, could be that you're running the op via CPU

moonbi 🇪🇺🪢 的头像

moonbi 🇪🇺🪢1 年前

The problem is u need huge gpu till run it

Cybersphere AI 的头像

Cybersphere AI1 年前

Setup is a nightmare on Windows. The setup is: FUSE-based (horribly slow) Assumes Linux KVM is present (Not true on Windows) Not built with Windows users in mind (requiring modifications to scripts) The devs need to release a setup procedure optimised for Windows.

Shabah 的头像

Shabah1 年前

This is really interesting

Electe 的头像

Electe1 年前

@huggingface @huggingface, incredible improvements with OmniParser. Speed like this can truly enhance workflows and drive productivity. 🚀 #Innovation

Tenkaizen 的头像

Tenkaizen1 年前

That's impressive progress. Faster parsing opens up so many possibilities for UI agents

Korey 🧘🏾‍♂️ 的头像

Korey 🧘🏾‍♂️1 年前

such a project, want to build some code samples with this.

相关视频

Microsoft released a groundbreaking model that can be used for web automation, with MIT license 🔥👏 OmniParser is a state-of-the-art UI parsing/understanding model that outperforms GPT4V in parsing. 👏

Microsoft released a groundbreaking model that can be used for web automation, with MIT license 🔥👏 OmniParser is a state-of-the-art UI parsing/understanding model that outperforms GPT4V in parsing. 👏

merve

473,157 次观看 • 1 年前

Microsoft just released an impressive tool OmniParser V2 can turn any LLM into an agent capable of using a computer 🔥 You can enable GPT-4o, DeepSeek R1, Sonnet 3.5, Qwen... to understand what's on your screen and take actions. 100% free & open source

Microsoft just released an impressive tool OmniParser V2 can turn any LLM into an agent capable of using a computer 🔥 You can enable GPT-4o, DeepSeek R1, Sonnet 3.5, Qwen... to understand what's on your screen and take actions. 100% free & open source

Paul Couvert

459,967 次观看 • 1 年前

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

OmniParser, the new screen parsing tool from Microsoft (and #1 trending model on Hugging Face), can now run 100% locally in your browser with Transformers.js! 🤯 Who's going to be the first to turn this into a browser extension? 👀 Endless possibilities! Demo & code below! 👇

Xenova

64,553 次观看 • 1 年前

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

I took Clicky and I made it 5x Faster So I saw that Farza 🇵🇰🇺🇸 was using uses Claude's vision to find UI elements on screen, send a screenshot, wait for coordinates back. It works, but it's slow. I replaced that with OmniParser V2 by Microsoft which is a local Object detection model trained specifically on UI elements. It runs on-device, detects every button, menu, and icon in 400ms, and gives me pixel-precise coordinates. No API call, no latency, no cost. The green highlights you see around the UI elements is the detection overlay and you can see as I am switching the screen it takes no time to detect and highlight which is pretty neat! With Local models improving by the day, Its the right direction for applications like these. Next up: video-synced tutorials where a YouTube tutorial pauses and waits for you to perform each action in the real app. I am not stopping!

Milind S

73,026 次观看 • 2 个月前

AG-UI makes building agentic applications dramatically easier. Here's how it works. This is a model for a simple chatbot: User → LLM → Response But interactive agents that render UI, pause for approvals, and ask users for input need a much more complex model. When building these agents, a response from the LLM will include a series of state changes as the agent runs: • Agent started a task • Agent called a tool • Agent updated its state • Agent streams these tokens • Agent is waiting on a human • Agent is resuming the task The Agent-User Interaction Protocol (AG-UI) treats the LLM response as a stream of events rather than a text endpoint. In practice, here is what you get as an agent runs: 1. Lifecycle events so your UI knows where the agent is. 2. Text messages that stream tokens. 3. Tool calls so your UI can prefill a form with any required arguments. 4. State updates that keep your UI in sync with the agent. 5. Special events for human approvals, rich media, and custom needs. All of these events travel over standard transports (SSE, WebSockets, or plain HTTP) as JSON. As a result, you can build a frontend that stays in sync with the agent's progress without having to invent a custom process to make this happen. For example, building a human-in-the-loop workflow becomes an off-the-shelf component you can integrate rather than build from scratch. CopilotKit🪁 is the creator of AG-UI, and you can use it when building frontend applications pretty much anywhere: • React • Angular • Vue • React Native • Slack • Teams • Discord • WhatsApp • Telegram Here is the link for you to check it out: Thanks to the CopilotKit team for partnering with me on this post.

AG-UI makes building agentic applications dramatically easier. Here's how it works. This is a model for a simple chatbot: User → LLM → Response But interactive agents that render UI, pause for approvals, and ask users for input need a much more complex model. When building these agents, a response from the LLM will include a series of state changes as the agent runs: • Agent started a task • Agent called a tool • Agent updated its state • Agent streams these tokens • Agent is waiting on a human • Agent is resuming the task The Agent-User Interaction Protocol (AG-UI) treats the LLM response as a stream of events rather than a text endpoint. In practice, here is what you get as an agent runs: 1. Lifecycle events so your UI knows where the agent is. 2. Text messages that stream tokens. 3. Tool calls so your UI can prefill a form with any required arguments. 4. State updates that keep your UI in sync with the agent. 5. Special events for human approvals, rich media, and custom needs. All of these events travel over standard transports (SSE, WebSockets, or plain HTTP) as JSON. As a result, you can build a frontend that stays in sync with the agent's progress without having to invent a custom process to make this happen. For example, building a human-in-the-loop workflow becomes an off-the-shelf component you can integrate rather than build from scratch. CopilotKit🪁 is the creator of AG-UI, and you can use it when building frontend applications pretty much anywhere: • React • Angular • Vue • React Native • Slack • Teams • Discord • WhatsApp • Telegram Here is the link for you to check it out: Thanks to the CopilotKit team for partnering with me on this post.

Santiago

17,438 次观看 • 10 天前

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,013 次观看 • 1 个月前

Introducing the Generative UI Research Canvas ✨ Every agent needs a powerful UI Combine LangChain, Tavily, Tako and CopilotKit🪁 and you get a LangChain-based research agent with a Generative UI-driven frontend. When agents can explore the underlying data, research becomes more trustworthy and useful. How does this work? - LangChain manages the agent workflow and tool orchestration. - Tako converts retrieved data into structured, visual components - Tavily provides agentic search - CopilotKit renders those components in the UI and keeps the user in the loop while the agent is working So instead of a boring and static report, the output is an interactive research canvas that evolves as the agent runs. 2026 is the year where users want to see the output, not read it... Generative UI makes this possible for agentic applications. Tutorial: Open-Source Repo: Docs:

Introducing the Generative UI Research Canvas ✨ Every agent needs a powerful UI Combine LangChain, Tavily, Tako and CopilotKit🪁 and you get a LangChain-based research agent with a Generative UI-driven frontend. When agents can explore the underlying data, research becomes more trustworthy and useful. How does this work? - LangChain manages the agent workflow and tool orchestration. - Tako converts retrieved data into structured, visual components - Tavily provides agentic search - CopilotKit renders those components in the UI and keeps the user in the loop while the agent is working So instead of a boring and static report, the output is an interactive research canvas that evolves as the agent runs. 2026 is the year where users want to see the output, not read it... Generative UI makes this possible for agentic applications. Tutorial: Open-Source Repo: Docs:

Atai Barkai

11,997 次观看 • 5 个月前

Microsoft Teams just got Generative UI But this is just the tip of the iceberg. AG-UI is the protocol connecting any agent to any user on any app, topped with Generative UI. That means your agent can render: - forms - buttons - charts - approval flows Right inside the chat, on the fly. But it's not limited to in-app copilots. What's special is that you can work with any agent using the same system and bring them into any surface, including Slack, WhatsApp, Discord and more. This is exactly why Hermes and OpenClaw took off. As users of these agents, we don't like walls of text, and it's a pain to get when you can have way better UX. CopilotKit🪁 + AG-UI handle that interaction layer. If you want to try out Agentic UI in Microsoft Teams, you can get early access here:

Microsoft Teams just got Generative UI But this is just the tip of the iceberg. AG-UI is the protocol connecting any agent to any user on any app, topped with Generative UI. That means your agent can render: - forms - buttons - charts - approval flows Right inside the chat, on the fly. But it's not limited to in-app copilots. What's special is that you can work with any agent using the same system and bring them into any surface, including Slack, WhatsApp, Discord and more. This is exactly why Hermes and OpenClaw took off. As users of these agents, we don't like walls of text, and it's a pain to get when you can have way better UX. CopilotKit🪁 + AG-UI handle that interaction layer. If you want to try out Agentic UI in Microsoft Teams, you can get early access here:

Atai Barkai

23,776 次观看 • 24 天前

Introducing Shadify: Generative UI built on ShadCN Simply describe a UI and watch your LangChain agent compose from @ShadCN on the fly, using AG-UI and CopilotKit🪁. Then export it as React code. Repo below. Try it out here:

Introducing Shadify: Generative UI built on ShadCN Simply describe a UI and watch your LangChain agent compose from @ShadCN on the fly, using AG-UI and CopilotKit🪁. Then export it as React code. Repo below. Try it out here:

Atai Barkai

138,046 次观看 • 3 个月前

From screenshot to functional UI in no time. Powered by Gemini 3 Flash, Antigravity can reproduce mockups with high fidelity. Just drop a screenshot, ask to rebuild as a Flutter UI, and watch it launch on your device.

From screenshot to functional UI in no time. Powered by Gemini 3 Flash, Antigravity can reproduce mockups with high fidelity. Just drop a screenshot, ask to rebuild as a Flutter UI, and watch it launch on your device.

Google Antigravity

394,843 次观看 • 5 个月前

How does Netflix know when you try to take a screenshot? Turns out it doesn’t. Rather than a simple piece of software recognizing a screenshot attempt and blacking out the screen, what’s going on is actually happening at the hardware level.

How does Netflix know when you try to take a screenshot? Turns out it doesn’t. Rather than a simple piece of software recognizing a screenshot attempt and blacking out the screen, what’s going on is actually happening at the hardware level.

tech.explain

487,149 次观看 • 2 个月前

Let's give this another go... 👀 Latest comparison of submenu performance for Base UI vs. bazza/ui <𝙼𝚎𝚗𝚞>. Updated to v1.2.0 — I was told v1.0.0 contained a temporary regression of this exact functionality. Base UI feels faster than before ⚡ Let me know what you think.

Let's give this another go... 👀 Latest comparison of submenu performance for Base UI vs. bazza/ui <𝙼𝚎𝚗𝚞>. Updated to v1.2.0 — I was told v1.0.0 contained a temporary regression of this exact functionality. Base UI feels faster than before ⚡ Let me know what you think.

kian bazza

39,105 次观看 • 4 个月前

We are witnessing a fundamental shift in how software is built. Over the decades, UI was static and predictable. But AI is changing the fundamental nature of the interface. We are moving away from fixed UI and toward a new standard: Generative UI. Generative UI lets agents participate in the UI, not just the conversation. It’s a dynamic participant in the user experience. Depending on the complexity of your app, Generative UI generally falls into three categories: 1. Static (AG-UI): UI is chosen from a fixed set of hand-built components 2. Open-Ended (MCP Apps): agent sends arbitrary content like raw HTML or iframes for total flexibility. 3. Declarative (A2UI): agent and frontend communicate via a structured UI spec for dynamic widgets and forms. The AG-UI Protocol by CopilotKit is the real breakthrough. AG-UI acts as the universal transport layer. It provides the integrations and middleware for every Gen-UI spec, making it trivial to bring dynamic components into your existing app. CopilotKit provides the go-to ecosystem for building agentic applications and is the React client of AG-UI. It is designed to be deeply integrated, allowing developers to bring their own frameworks or protocols to the table. Play around with different types of Generative UI here → Check out the video attached to see Generative UI in action. Appreciate the CopilotKit team for collaborating with me on this deep dive.

We are witnessing a fundamental shift in how software is built. Over the decades, UI was static and predictable. But AI is changing the fundamental nature of the interface. We are moving away from fixed UI and toward a new standard: Generative UI. Generative UI lets agents participate in the UI, not just the conversation. It’s a dynamic participant in the user experience. Depending on the complexity of your app, Generative UI generally falls into three categories: 1. Static (AG-UI): UI is chosen from a fixed set of hand-built components 2. Open-Ended (MCP Apps): agent sends arbitrary content like raw HTML or iframes for total flexibility. 3. Declarative (A2UI): agent and frontend communicate via a structured UI spec for dynamic widgets and forms. The AG-UI Protocol by CopilotKit is the real breakthrough. AG-UI acts as the universal transport layer. It provides the integrations and middleware for every Gen-UI spec, making it trivial to bring dynamic components into your existing app. CopilotKit provides the go-to ecosystem for building agentic applications and is the React client of AG-UI. It is designed to be deeply integrated, allowing developers to bring their own frameworks or protocols to the table. Play around with different types of Generative UI here → Check out the video attached to see Generative UI in action. Appreciate the CopilotKit team for collaborating with me on this deep dive.

Pratham

33,920 次观看 • 5 个月前

I hated spending minutes trying to explain which element I meant in a long prompt, so I shipped something to fix it. Inspector’s new precision UI select tool connects the element on your screen to its exact source code, saving you time on prompting.

I hated spending minutes trying to explain which element I meant in a long prompt, so I shipped something to fix it. Inspector’s new precision UI select tool connects the element on your screen to its exact source code, saving you time on prompting.

mikeBuildsMore

12,343 次观看 • 8 个月前

This Claude 3.7 UI workflow is awesome And many vibe coders not aware of it: 1. Have Claude 3.7 to design whole UI screen by screen 2. Ask it to make a plan, break down into components 3. Turn into actual project By having Claude focus on just UI generation, it does a great job You can build beautiful UI without shadcn If you want to learn more about vibe coding beyond basics, join my community here: (also get free access to 10xCoder.Dev)

This Claude 3.7 UI workflow is awesome And many vibe coders not aware of it: 1. Have Claude 3.7 to design whole UI screen by screen 2. Ask it to make a plan, break down into components 3. Turn into actual project By having Claude focus on just UI generation, it does a great job You can build beautiful UI without shadcn If you want to learn more about vibe coding beyond basics, join my community here: (also get free access to 10xCoder.Dev)

Jason Zhou

745,638 次观看 • 1 年前

Microsoft 🤝 AG-UI Protocol Microsoft Agent Framework is now AG-UI compatible! 🎉 Connect your MS Agent Framework .NET agents into fullstack applications with Python coming soon 👀 MS Agent Framework takes care of the agentic backend, AG-UI connects your agents to the frontend. Features include: - Chat UI - Generative UI - State synchronization - Human-in-the-loop - Frontend Tool Calls All the building blocks you need to take your agents from the backend and into a fullstack agentic application. Available in React & Angular via CopilotKit, and Rust, Kotlin, Rust, Java and .NET via community SDKs Big shoutout to the Microsoft & CopilotKit teams, who shipped this highly requested integration at breakneck speeds. More coming soon 👀 🙏 Javier Nelson, Joonseok Oh, Daniel Roth, Evan Mattson, Tyler Slaton, Brandon McConnell, Markus Ecker Read more about our collaboration: Get started: Check out the Dojo:

Microsoft 🤝 AG-UI Protocol Microsoft Agent Framework is now AG-UI compatible! 🎉 Connect your MS Agent Framework .NET agents into fullstack applications with Python coming soon 👀 MS Agent Framework takes care of the agentic backend, AG-UI connects your agents to the frontend. Features include: - Chat UI - Generative UI - State synchronization - Human-in-the-loop - Frontend Tool Calls All the building blocks you need to take your agents from the backend and into a fullstack agentic application. Available in React & Angular via CopilotKit, and Rust, Kotlin, Rust, Java and .NET via community SDKs Big shoutout to the Microsoft & CopilotKit teams, who shipped this highly requested integration at breakneck speeds. More coming soon 👀 🙏 Javier Nelson, Joonseok Oh, Daniel Roth, Evan Mattson, Tyler Slaton, Brandon McConnell, Markus Ecker Read more about our collaboration: Get started: Check out the Dojo:

CopilotKit🪁

40,520 次观看 • 8 个月前

Generative UI will be the next big shift in AI. Where your product UI is dynamically generated, based on user intent and context. If you are a developer building AI apps, you need to hear this out:

Generative UI will be the next big shift in AI. Where your product UI is dynamically generated, based on user intent and context. If you are a developer building AI apps, you need to hear this out:

Pratham

119,196 次观看 • 8 个月前

🎨 Design freely with Agent 4. - Design while the agent builds. - Explore multiple UI variants on an infinite canvas. - Make precise UI edits that you can apply directly to your app.

🎨 Design freely with Agent 4. - Design while the agent builds. - Explore multiple UI variants on an infinite canvas. - Make precise UI edits that you can apply directly to your app.

Replit ⠕

184,057 次观看 • 3 个月前

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Hugo

11,671 次观看 • 2 个月前

i built the first ever email to webhook platform sub 300ms processing, beautiful UI and open source try it for free below

i built the first ever email to webhook platform sub 300ms processing, beautiful UI and open source try it for free below

ryan vogel

99,642 次观看 • 1 年前