Loading video...

Video Failed to Load

Go Home

Ensure your coding agent creates more accessible, progressively enhanced web forms, with Modern Web Guidance. Watch our side-by-side comparison to discover how Modern Web Guidance skills help agents handle: ⌨️ Adding the Enter key hint attribute 🏷️ Correct autocomplete values 🎨 Autofill styling as a progressive enhancement See it...

13,707 views • 11 days ago •via X (Twitter)

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

New Short Course: Building AI Browser Agents! Learn how to build AI agents that interact and take actions on websites in this course, created in partnership with and taught by and @namangarg0, Co-founders of AGI Inc. AI browser agents can log into websites, fill out forms, click through web pages, or even place orders online for you. They use both visual information, like screenshots, and structural data, like the HTML or Document Object Model (DOM) of a web page, to reason and take action. With the complexity of webpages and multiple possible actions at each step, it can be challenging for an AI browser agent to complete an assigned task. Because these agents run long action sequences, a single error—like clicking the wrong button or misreading a field—can lead to unexpected outcomes or errors that compound over time. In this course, you'll understand how autonomous web agents work, their current limitations, and how AgentQ enables them to improve through self-correction. In detail, you'll: - Learn what web agents are, how they automate tasks online, their architecture, key components, limitations, and an overview of their decision-making strategies. - Build a web agent that can scrape website and return course recommendations in a structured output format. - Build an autonomous web agent that can execute multiple tasks, such as finding and summarizing webpages, filling out a form, and signing up for a newsletter. - Explore AgentQ, a framework that enables agents to self-correct by combining Monte Carlo Tree Search (MCTS), a self-critique mechanism for continuous improvement, and Direct Preference Optimization (DPO). - Deep dive into MCTS, learn how it finds an effective path, illustrated by an example of Gridworld animation, and use AgentQ to complete web tasks. - Understand AI agents' current state and future directions—including key factors shaping their evolution, such as hardware, algorithm innovation, and data availability. By the end of this course, you will have hands-on experience building browser agents and a deeper understanding of how to make them more robust and reliable. Please sign up here:

Andrew Ng

186,182 views • 1 year ago

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,697 views • 2 years ago

New course to bring you up to state-of-the-art at using AI to help you code: Build Apps with Windsurf's AI Coding Agents, built in partnership with WIndsurf (Codeium) and taught by Anshul Ramachandran! AI-assisted IDEs (Integrated Development Environments) make developers’ workflows faster, more efficient, and much more fun. Agentic tools like Windsurf are more than just code autocomplete—they are collaborative coding agents that help you break down complex applications, iterate efficiently, and generate code that spans multiple files. Although a lot of coding assistants share the same underlying large language models for planning and reasoning, a major point of distinction is how they handle tools, keep track of context, and stay aligned with your intent as a developer. For instance, if you make modifications to a class definition in your code and make the same modifications to other classes in the same directory, you might tell the AI agent "Do the same thing in similar places in this directory." Here, tracking your intent means understanding that “the same thing" refers to that recent edit you just made, which must be followed by appropriate search and tool-calling to implement the changes. In this course, you'll learn the inner workings of coding agents, their strengths and limitations, and how to use Windsurf to quickly build several applications. In detail, you'll: - Build a mental model of how agents work by combining human-action tracking, tool integration, and context awareness to carry out an agentic coding workflow. - Learn the challenges of code search and discovery and how a multi-step retrieval approach helps coding agents address them. - Use Windsurf to analyze and understand a large, old codebase and update it to the latest versions of the frameworks and packages it uses. - Build a Wikipedia data analysis app that retrieves, parses, and analyzes word frequencies. - Enhance the performance of your Wikipedia analysis app by adding caching, and through this, also learn how to course-correct when the AI agent produces unexpected results. - Learn tips and tricks such as keyboard shortcuts, autocomplete, and @ mentions to quickly call on agentic capabilities. - Use image/multimodal capabilities of the AI agent to increase your development velocity; you'll see an example of uploading a mockup with sketched-out UI features, and ask the agent to use that to build new functionality to an app. By the end of this course, you’ll understand agentic coding in-depth and know how to use it to make your development process much faster, more efficient, and enjoyable. Please sign up here!

Andrew Ng

140,307 views • 1 year ago

What happens when there are more agents than eyeballs browsing the internet? How will the economics of the internet be reinvented? Parag Agrawal thinks Shapley values may hold the answer. Parag built Twitter over a decade, eventually becoming CEO and selling the company to Elon. He's spent the last three years building Parallel Web Systems, a search engine built for agents instead of humans. His core argument: 1) human click data is a bug. Agents aren't just a new technology, they're a distinct customer – and the feedback loop that made Google great is the wrong signal for the thing actually doing the work; 2) the ad-funded web assumed scarce human attention. If agents show up instead of eyeballs, the business model underneath the internet has to be rebuilt, or good content stops being published. The conversation covers: — why he shipped a search agent before a search engine, and how that let him grow the index incrementally instead of buying a full web crawl up front — the billion-to-billion matching problem: pull the right 1,000 tokens out of a trillion web pages, and your agent uses under half the tokens — going from a 3-second compute budget to 200 milliseconds — why he doesn't think Parallel is a neo-lab: "our output is a complement to a model" — the Google Cloud deal — on GCP, your grounding options are now Google Search or Parallel Search — reinventing the economics of the internet, Shapley values as a payment rail for publishers, and why the company was originally incorporated as Shapley Inc — why routing 2-10% of inference spend to web data would dwarf every content business outside the walled gardens — the web going from pull to push: "call me if this happens" 00:00 Introduction 03:25 What Is Web Search 05:17 Why Start a New Index 07:52 Search Agents First 10:17 Not a Neolab 13:14 Agents vs Google Search 19:38 Inside the Search Stack 28:59 Search Multipliers With Agents 30:21 Meeting Prep Agent Workflows 31:46 Quality Cost Latency And Turbo 32:42 Are Agents Overtaking Humans 34:28 Ads Model Meets Agent Web 37:20 New Incentives For Content 40:48 Shapley Values Attribution 47:46 Parallel Web And Future Vision Hosted with my very unwilling co-host Andrew Reed and Sequoia Capital

Sonya Huang 🐥

129,143 views • 1 month ago

how to use firecrawl to give your AI eyes and actually build startups that outperform 99% of apps: 1. your AI is smart but blind. it can't go to a website, read a page, or grab data on its own. firecrawl fixes that. you put in a URL. you get back clean markdown, structured JSON, screenshots. feed it to any model. 2. three lines of code. that's it. no proxies. no anti-bot detection. no custom scrapers that break when a site changes. one API call. clean data back in seconds. works on 98%+ of sites. 3. firecrawl has six core capabilities: scrape a single page. crawl an entire site. map all URLs on a domain. search google and return full content. an agent endpoint where you describe what you want and it goes and finds it. and a browser sandbox where AI controls a real browser like filling forms, clicking buttons, handles logins. 4. the agent endpoint is wild. you can say "find all of YC's winter 24 dev tool companies and their founders and emails" and get back structured data. or "compare pricing tiers across stripe, square, and paypal" and get a side-by-side table. 5. the browser sandbox lets your AI stay logged in across sessions, navigate pagination, watch live as it browses. this is computer use without building the infrastructure yourself. 6. think of it in layers. every builder needs: an agent harness (claude code, cursor, codex), a search layer (perplexity, exa), a web data layer (firecrawl), an ops brain (obsidian, notion), and an outbound stack. the web data layer is the one most people are sleeping on. 7. this is the AWS moment for web data. in 2006 building a web app meant buying servers and managing racks. AWS said one API call, use our servers. some of the biggest companies of the last decade were built on that. firecrawl is doing the same thing for web data in 2026. 8. the framework i'd use for coming up with startup ideas building with clean data: take a massive horizontal platform. rebuild it for one niche using firecrawl. the vertical version always wins because people want specific, not generic. price for outcome. 9. a year ago firecrawl posted a job listing that said "please only apply if you're an AI agent." content creator agents. customer support agents. junior dev agents. it looked weird. it was a signal for where this is all going. the people who understand how to get clean web data, wrap it around an LLM, and package it as a product are the the ones with a 12-month head start. i use Firecrawl with Idea Browser . once you see what's possible with structured web data, you can't unsee it. episode is live on The Startup Ideas Podcast (SIP) 🧃 (full breakdown there) i tried to explain this as clear as possible for even the non technical. send it to a builder friend. watch

GREG ISENBERG

135,254 views • 6 months ago