Introducing llms.txt Generator ✨ You can now concatenate any website into a single text file that can be fed into any LLM. We crawl the whole website with and extract data with gpt-4o-mini. Create your own llms.txt at

You can generate both llms.txt and llms-full.txt files through the web UI or by sending a GET request to There is no API key required but with a free Firecrawl API key you have no limit. Due to the LLM calls it may take up to a few minutes.

You can learn more about the llms.txt standard at It is a great idea by @jeremyphoward & @HamelHusain!

There’s also an awesome directory of llms.txt adopters at built by @iFox & @joyceverheije! Would love to integrate llms.txt generator!

Also this is completely open source if anyone wants to improve it:

@firecrawl_dev You can check out my too. I actually launched it first, working on the update rn.

@firecrawl_dev Sweet!!

@firecrawl_dev Great tool

@firecrawl_dev Thank you!

@firecrawl_dev You crazy. I'm on it already

@firecrawl_dev I tried this on a non-tech website and was able to get Claude to artificially navigate the website (based on 15 personas with objectives) for testing navigation and information architecture. Very cool.