Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

OpenAI's Swarm Web Extractor This can autonomously search the web, map entire websites, and extract data. This is built on top of OpenAI’s new multi-agent framework Swarm, Serp AI and Firecrawl API. - Swarm is a lightweight and experimental framework introduced by OpenAI to develop multi-agent systems. - Serp... show more

Kalyan KS

3,622 subscribers

174,419 views • 1 year ago •via X (Twitter)

News & Politics Science & Technology Education

Anya Rossi• Live Now

Private livecam show

10 Comments

NICE1 year ago

Stay competitive by balancing cutting-edge AI with automation tools. Forrester shows how.

Firecrawl1 year ago

🔥🔥🔥

OG Whippersnapper1 year ago

They're automating the Semantic Web. If they scrape, augment, and store it's going to make for a hell of a lot of solid training data. Not to mention the foundation of a better web search.

Alexander De Ridder1 year ago

You call it mapping and extracting; I see an invasion of privacy. Data ethics should guide innovation, not vice versa.

Robin And The Dog ⌘ AICoding1 year ago

So AI is build on stolen web data, now can steal even more with an agent lol.

Colleen M. Brady1 year ago

Thanks for sharing. 🙏 I prototyped something very rudimentary along these lines over the weekend. Was trying to decide how to rebuild it and take it to the next level. This looks like a much better starting point.

John E-gen 🔮1 year ago

went thru their code base you wouldn’t happen to have that swarm.agents code package (?) , i see the import but not the code

Zafar Kahn 🇺🇸1 year ago

Nice

•UtilityCo1 year ago

Need more of this in my feed. Thank you, Kalyanji.

Tobsn1 year ago

but what’s the cost point per scrape vs just using code to extract data?

Related Videos

Firecrawl open-sourced "web-agent," a new framework that allows developers to build AI agents capable of autonomously executing "search-scrape-interact" loops on the live web. By running a single command, developers can scaffold a complete project with streaming UI, API, or library templates. The framework uses a "Plan-Act" mechanism and can spin up parallel sub-agents to process multiple websites simultaneously. It is entirely model-agnostic, allowing you to plug in OpenAI, Anthropic (with recent builds defaulting to Claude Opus 4.7), or your own self-hosted models.

Firecrawl open-sourced "web-agent," a new framework that allows developers to build AI agents capable of autonomously executing "search-scrape-interact" loops on the live web. By running a single command, developers can scaffold a complete project with streaming UI, API, or library templates. The framework uses a "Plan-Act" mechanism and can spin up parallel sub-agents to process multiple websites simultaneously. It is entirely model-agnostic, allowing you to plug in OpenAI, Anthropic (with recent builds defaulting to Claude Opus 4.7), or your own self-hosted models.

Wes Roth

53,653 views • 4 months ago

Introducing /search - the simplest way for agents and devs to discover the web 👀 Our #1 most requested endpoint is finally here. Search the web AND scrape all results in an LLM-ready format with one API call. Now live on our API, MCP, and all integrations 🔥

Introducing /search - the simplest way for agents and devs to discover the web 👀 Our #1 most requested endpoint is finally here. Search the web AND scrape all results in an LLM-ready format with one API call. Now live on our API, MCP, and all integrations 🔥

Firecrawl

390,772 views • 1 year ago

Agents can now search and scrape the web with a single API call. Firecrawl's /search endpoint returns complete LLM-ready content for every search result. HTML, Markdown, or JSON. Works via API, MCP for Cursor, Zapier and n8n.

Agents can now search and scrape the web with a single API call. Firecrawl's /search endpoint returns complete LLM-ready content for every search result. HTML, Markdown, or JSON. Works via API, MCP for Cursor, Zapier and n8n.

Lior Alexander

14,812 views • 1 year ago

Introducing /agent by Firecrawl 🪄 Just describe what you need - with or without a URL then /agent searches, navigates, and gathers information from the widest range of websites, reaching data no other API can. Try out the research preview today.

Introducing /agent by Firecrawl 🪄 Just describe what you need - with or without a URL then /agent searches, navigates, and gathers information from the widest range of websites, reaching data no other API can. Try out the research preview today.

Firecrawl

613,410 views • 8 months ago

Oh wow…Firecrawl just dropped /agent, an API that autonomously navigates and discovers data from any website. You tell it what data you need, it figures out how to find it across complex sites. This is the future of web data.

Oh wow…Firecrawl just dropped /agent, an API that autonomously navigates and discovers data from any website. You tell it what data you need, it figures out how to find it across complex sites. This is the future of web data.

Alvaro Cintas

159,050 views • 8 months ago

I built a Deep Research AI Agent team using OpenAI Agents SDK and Firecrawl. It combines multiple AI agents that can autonomously search the web, extract content, and generate detailed reports with enhanced analysis. 100% Opensource Code with step-by-step tutorials.

I built a Deep Research AI Agent team using OpenAI Agents SDK and Firecrawl. It combines multiple AI agents that can autonomously search the web, extract content, and generate detailed reports with enhanced analysis. 100% Opensource Code with step-by-step tutorials.

Shubham Saboo

50,831 views • 1 year ago

.Riveter is replacing the entire scraper infrastructure stack with a single API. Their all-in-one search agent replaces SERP, scrapers, proxy managers, and more. And it knows which tools to use to find exactly the data you’re looking for. Congrats on the launch, Abby Grills and cody!

.Riveter is replacing the entire scraper infrastructure stack with a single API. Their all-in-one search agent replaces SERP, scrapers, proxy managers, and more. And it knows which tools to use to find exactly the data you’re looking for. Congrats on the launch, Abby Grills and cody!

Y Combinator

23,992 views • 10 months ago

Firecrawl is now a search provider in OpenRouter 🔥 Ground any OpenRouter model in full web pages instead of search snippets. Firecrawl searches and scrapes the live web into clean markdown your model can reason over. New sign-ups get a free Hobby plan with 100,000 credits!

Firecrawl is now a search provider in OpenRouter 🔥 Ground any OpenRouter model in full web pages instead of search snippets. Firecrawl searches and scrapes the live web into clean markdown your model can reason over. New sign-ups get a free Hobby plan with 100,000 credits!

Firecrawl

30,108 views • 4 months ago

how to use firecrawl to give your AI eyes and actually build startups that outperform 99% of apps: 1. your AI is smart but blind. it can't go to a website, read a page, or grab data on its own. firecrawl fixes that. you put in a URL. you get back clean markdown, structured JSON, screenshots. feed it to any model. 2. three lines of code. that's it. no proxies. no anti-bot detection. no custom scrapers that break when a site changes. one API call. clean data back in seconds. works on 98%+ of sites. 3. firecrawl has six core capabilities: scrape a single page. crawl an entire site. map all URLs on a domain. search google and return full content. an agent endpoint where you describe what you want and it goes and finds it. and a browser sandbox where AI controls a real browser like filling forms, clicking buttons, handles logins. 4. the agent endpoint is wild. you can say "find all of YC's winter 24 dev tool companies and their founders and emails" and get back structured data. or "compare pricing tiers across stripe, square, and paypal" and get a side-by-side table. 5. the browser sandbox lets your AI stay logged in across sessions, navigate pagination, watch live as it browses. this is computer use without building the infrastructure yourself. 6. think of it in layers. every builder needs: an agent harness (claude code, cursor, codex), a search layer (perplexity, exa), a web data layer (firecrawl), an ops brain (obsidian, notion), and an outbound stack. the web data layer is the one most people are sleeping on. 7. this is the AWS moment for web data. in 2006 building a web app meant buying servers and managing racks. AWS said one API call, use our servers. some of the biggest companies of the last decade were built on that. firecrawl is doing the same thing for web data in 2026. 8. the framework i'd use for coming up with startup ideas building with clean data: take a massive horizontal platform. rebuild it for one niche using firecrawl. the vertical version always wins because people want specific, not generic. price for outcome. 9. a year ago firecrawl posted a job listing that said "please only apply if you're an AI agent." content creator agents. customer support agents. junior dev agents. it looked weird. it was a signal for where this is all going. the people who understand how to get clean web data, wrap it around an LLM, and package it as a product are the the ones with a 12-month head start. i use Firecrawl with Idea Browser . once you see what's possible with structured web data, you can't unsee it. episode is live on The Startup Ideas Podcast (SIP) 🧃 (full breakdown there) i tried to explain this as clear as possible for even the non technical. send it to a builder friend. watch

how to use firecrawl to give your AI eyes and actually build startups that outperform 99% of apps: 1. your AI is smart but blind. it can't go to a website, read a page, or grab data on its own. firecrawl fixes that. you put in a URL. you get back clean markdown, structured JSON, screenshots. feed it to any model. 2. three lines of code. that's it. no proxies. no anti-bot detection. no custom scrapers that break when a site changes. one API call. clean data back in seconds. works on 98%+ of sites. 3. firecrawl has six core capabilities: scrape a single page. crawl an entire site. map all URLs on a domain. search google and return full content. an agent endpoint where you describe what you want and it goes and finds it. and a browser sandbox where AI controls a real browser like filling forms, clicking buttons, handles logins. 4. the agent endpoint is wild. you can say "find all of YC's winter 24 dev tool companies and their founders and emails" and get back structured data. or "compare pricing tiers across stripe, square, and paypal" and get a side-by-side table. 5. the browser sandbox lets your AI stay logged in across sessions, navigate pagination, watch live as it browses. this is computer use without building the infrastructure yourself. 6. think of it in layers. every builder needs: an agent harness (claude code, cursor, codex), a search layer (perplexity, exa), a web data layer (firecrawl), an ops brain (obsidian, notion), and an outbound stack. the web data layer is the one most people are sleeping on. 7. this is the AWS moment for web data. in 2006 building a web app meant buying servers and managing racks. AWS said one API call, use our servers. some of the biggest companies of the last decade were built on that. firecrawl is doing the same thing for web data in 2026. 8. the framework i'd use for coming up with startup ideas building with clean data: take a massive horizontal platform. rebuild it for one niche using firecrawl. the vertical version always wins because people want specific, not generic. price for outcome. 9. a year ago firecrawl posted a job listing that said "please only apply if you're an AI agent." content creator agents. customer support agents. junior dev agents. it looked weird. it was a signal for where this is all going. the people who understand how to get clean web data, wrap it around an LLM, and package it as a product are the the ones with a 12-month head start. i use Firecrawl with Idea Browser . once you see what's possible with structured web data, you can't unsee it. episode is live on The Startup Ideas Podcast (SIP) 🧃 (full breakdown there) i tried to explain this as clear as possible for even the non technical. send it to a builder friend. watch

GREG ISENBERG

135,254 views • 5 months ago

This is the DeepSeek moment of Search? A small team of cracked engineers built a search API for AI agents that outperforms everything built with 20x the funding. AI systems are about to become the biggest users of information on Earth. They don’t need a better Google. They need a retrieval layer built for reasoning, scale, and access beyond the web. That’s what we’re building. 1/3

This is the DeepSeek moment of Search? A small team of cracked engineers built a search API for AI agents that outperforms everything built with 20x the funding. AI systems are about to become the biggest users of information on Earth. They don’t need a better Google. They need a retrieval layer built for reasoning, scale, and access beyond the web. That’s what we’re building. 1/3

Valyu

1,135,072 views • 10 months ago

Smooth by Smooth is a real-time AI browser agent - and the fastest and most reliable one yet. It can autonomously perform any task on the web using mouse and keyboard. Infinitely scalable, production-ready, and secure, it's now available as a serverless API with instant browser spin-up and persistent sessions.

Smooth by Smooth is a real-time AI browser agent - and the fastest and most reliable one yet. It can autonomously perform any task on the web using mouse and keyboard. Infinitely scalable, production-ready, and secure, it's now available as a serverless API with instant browser spin-up and persistent sessions.

Y Combinator

33,958 views • 10 months ago

French first, English below 👇 🇪🇺 Staan, la première API de recherche européenne est désormais disponible en libre-service 🇪🇺 Staan est l'API de recherche 100% européenne co-construite avec Ecosia à travers la joint-venture européenne European Search Perspective. Staan connecte les applications d'IA au web en les alimentant avec des millions de documents et de contenus web dans un cadre juridique respectueux des données et des utilisateurs. À partir d'aujourd'hui, Staan permet à n'importe quel développeur d'obtenir sa clé API et de commencer à alimenter ses assistants, copilotes et agents IA en quelques minutes. Pendant que les solutions américaines restreignent leurs API de recherche et leurs modèles de LLM, une seconde voie européenne et souveraine s'organise. ✊ Switch to Europe. Now. ⸻ 🇪🇺 Staan, the first European search API, is now available as a self-service 🇪🇺 Staan is the 100% European search API co-built with Ecosia through the European joint venture European Search Perspective. Staan connects AI applications to the web by feeding them with millions of web documents and content within a legal framework that respects data and users. Starting today, Staan allows any developer to get their API key and start powering their assistants, copilots and AI agents in minutes. While American solutions are restricting their search APIs and LLM models, a second European and sovereign path is taking shape. ✊ Switch to Europe. Now.

French first, English below 👇 🇪🇺 Staan, la première API de recherche européenne est désormais disponible en libre-service 🇪🇺 Staan est l'API de recherche 100% européenne co-construite avec Ecosia à travers la joint-venture européenne European Search Perspective. Staan connecte les applications d'IA au web en les alimentant avec des millions de documents et de contenus web dans un cadre juridique respectueux des données et des utilisateurs. À partir d'aujourd'hui, Staan permet à n'importe quel développeur d'obtenir sa clé API et de commencer à alimenter ses assistants, copilotes et agents IA en quelques minutes. Pendant que les solutions américaines restreignent leurs API de recherche et leurs modèles de LLM, une seconde voie européenne et souveraine s'organise. ✊ Switch to Europe. Now. ⸻ 🇪🇺 Staan, the first European search API, is now available as a self-service 🇪🇺 Staan is the 100% European search API co-built with Ecosia through the European joint venture European Search Perspective. Staan connects AI applications to the web by feeding them with millions of web documents and content within a legal framework that respects data and users. Starting today, Staan allows any developer to get their API key and start powering their assistants, copilots and AI agents in minutes. While American solutions are restricting their search APIs and LLM models, a second European and sovereign path is taking shape. ✊ Switch to Europe. Now.

Qwant

114,529 views • 2 months ago

Your agents suck when using the web because 85% of it doesn't have an API. Browserbase gives them everything they need to do work online. Leading AI companies like Ramp, Lovable, and Clay trust us to power agents that do real work on behalf of real people. With a single API key, your agent gets everything it needs to navigate the wild web: browsers, search, fetch, identity, a sandbox runtime, and model gateway. Stop waiting on integrations, build agents that can browse and interact with the web just like humans.

Your agents suck when using the web because 85% of it doesn't have an API. Browserbase gives them everything they need to do work online. Leading AI companies like Ramp, Lovable, and Clay trust us to power agents that do real work on behalf of real people. With a single API key, your agent gets everything it needs to navigate the wild web: browsers, search, fetch, identity, a sandbox runtime, and model gateway. Stop waiting on integrations, build agents that can browse and interact with the web just like humans.

Paul Klein IV

1,729,760 views • 4 months ago

Agent Mode in the @Geminiapp can help you get more done across the web - coming to subscribers soon. Plus a new multi-tasking version of Project Mariner is now available to Google AI Ultra subscribers in the US, and computer use capabilities are coming to the Gemini API.

Agent Mode in the @Geminiapp can help you get more done across the web - coming to subscribers soon. Plus a new multi-tasking version of Project Mariner is now available to Google AI Ultra subscribers in the US, and computer use capabilities are coming to the Gemini API.

Sundar Pichai

68,224 views • 1 year ago

OptimAI Search is now on Agentic(.)Market - and we’re bringing the stack to Base. Builders can plug into live, citation-backed context across web, social, and crypto data through MCP, API, and SDK - then pay seamlessly with x402. No brittle workflows. No manual stitching. Just agent-native access to real-time intelligence at scale. And with custom domain agents on the marketplace, specialized knowledge becomes reusable. Search is becoming infrastructure. Build with it: 💡 💡

OptimAI Search is now on Agentic(.)Market - and we’re bringing the stack to Base. Builders can plug into live, citation-backed context across web, social, and crypto data through MCP, API, and SDK - then pay seamlessly with x402. No brittle workflows. No manual stitching. Just agent-native access to real-time intelligence at scale. And with custom domain agents on the marketplace, specialized knowledge becomes reusable. Search is becoming infrastructure. Build with it: 💡 💡

OptimAI Network

63,832 views • 3 months ago

We’re officially a YOUnicorn! 🦄 Excited to share that You.com just raised $100M Series C at a $1.5B valuation, led by Cox Enterprises. We’ve been heads down building the search infrastructure for the AI and agent future. Soon there will be more AI agents using the web than humans, but today's search wasn't built for this. Agents need deep, contextual information from both public web and internal private data to make real decisions. Our web search API delivers the most up-to-date, accurate, and fastest search results for LLMs and agents. Real benchmarks show we consistently outperform the competition on accuracy and speed while staying cost-effective. Today, we’re answering over 1 billion queries every month across enterprises like DuckDuckGo, Windsurf, Harvey, and many others. There is no other startup of recent years at this scale. Grateful to our wonderful employees, customers, partners, and investors who believe in our vision. The agentic era is here - let's build it together. Try our LLM search APIs.

We’re officially a YOUnicorn! 🦄 Excited to share that You.com just raised $100M Series C at a $1.5B valuation, led by Cox Enterprises. We’ve been heads down building the search infrastructure for the AI and agent future. Soon there will be more AI agents using the web than humans, but today's search wasn't built for this. Agents need deep, contextual information from both public web and internal private data to make real decisions. Our web search API delivers the most up-to-date, accurate, and fastest search results for LLMs and agents. Real benchmarks show we consistently outperform the competition on accuracy and speed while staying cost-effective. Today, we’re answering over 1 billion queries every month across enterprises like DuckDuckGo, Windsurf, Harvey, and many others. There is no other startup of recent years at this scale. Grateful to our wonderful employees, customers, partners, and investors who believe in our vision. The agentic era is here - let's build it together. Try our LLM search APIs.

Richard Socher

63,660 views • 11 months ago