GPT-4 is amazing at scraping data from websites.

@Scobleizer Redefines the "purpose of a website" going forward no human may actually 'visit' a website anymore - at some point

Lol, now think about the price. 1000 tokens (say one page) costs $0.03. It means scraping only 10k pages will be $300 🤯. Instead of parsing, gpt4 can be used for writing the scraper code.

When I was doing something similar, I had a funnier way of getting the visible text from a page. Told Playwright to print to PDF, then read the raw text out of the PDF 😆 Amazingly with both Chrome's algorithms and many sites having print style sheets, it worked well!

Instead of using puppeteer and stripping html tags you could use something like ;) javascript support shipping soon.

For most tasks like this, GPT-3.5 will be able to provide similar quality output and will be 20x~ cheaper. And it is now even faster since you can show much more of a page to it at once with the new 16k model.

Does is crawl one page only or follow links? Also is it using the new function APIs inside the GPT runner?

I built 2 side projects based on its ability to scrape data 😆 They are 0 MRR, but I'm still hopeful.

Love this. Hard to find real business applications of folks using it other than for copy ideas.

what if the text size exceeds the context window?