Loading video...
Video Failed to Load
I've built it for you!! It's an automated AI system that analyzes AI case studies (you can change the use case) to identify and document enterprise-level AI implementations. It starts by reading URLs from a CSV file and uses web scraping (either through WebLoader or Firecrawl) to extract the... show more
85,221 views • 1 year ago •via X (Twitter)
11 Comments

I want to run AI agents to scrape specific URLs and do some data extraction until they find a certain kind of information. Like an AI investigator. What’s the framework that allows for this kind of cutting-edge stuff? Is there an AI agent project that we should be using?

The repo is available for anyone who wants to try it

Amazing!

I hope it helps. Thank you.

@arvidkahl Yeah CrawlAI or Fire crawl is definitely a good scraper to use for such a project. Fire crawl is seriously improving overtime.

@arvidkahl White IDE oh no, my eyes 😑

Looks awesome, I really would love to know what the total cost of use is for this system implementation. Say running it on let’s just say a weather website or API on a daily basis for a week for one single location. Or if that’s too simple, then another example would be the cost projections for running this type of system on an aggregate of meet up and technology focused events. Essentially just seems like an insane amount of API token and function calling. Use to where even just running it for an hour would cost probably close to 100 bucks USD?

@arvidkahl I don't think it would be costly if you use Haiku or any other smaller and cheaper model.

@arvidkahl that’s one horrendous theme

@arvidkahl Cool one Muratcan ! 👏

@arvidkahl This is intriguing but I’m not sure I’m clear on what problem this is solving. Can you help me understand?
