I built a bot traffic tracker for DataFast 🤖 It shows what crawlers and AI assistants do on your website: ✨ ChatGPT / Claude fetching pages to answer users 🔎 Google / Bing indexing your site 🧠 OpenAI / Anthropic / Meta crawling for training 📄 The exact pages... they crawl 🚫 Pages they try to crawl that return 404s The 404s are interesting because if ChatGPT keeps requesting /free-trial and that page does not exist, that is a signal you should build it! Free for early users, probably not forever. If you try it, I’d love feedback. PS. Let me know your bot/human traffic rate (mine is ~300%! 🤯)show more

Get started here:

I also built a public crawler directory you can use:

@DataFast_ user agent strings are spoofable, anyone can send GPTBot in a header. without bot auth or header verification youre just charting whatever they claim to be

@netrunner_btc @DataFast_ IP verification

@DataFast_ right, thats the actual fix not the ua string. was more pointing out the header alone means nothing

@DataFast_ Set this up as soon as I saw it on my dashboard - legendary! I think it would be good to extend the card to use the data to display a table underneath that showed the most requested urls etc, with a filter by the 404's.

@DataFast_ If you click on crawlers you’ll see these urls but you’re saying it’s bot enough and you want to see all urls for all crawlers?

@DataFast_ <3 thanks legend

@DataFast_ Yes already working

@DataFast_ Meta scrapping like crazy

@DataFast_ That’s just today!

@DataFast_ Hey Marc! team launched an agent for situational marketing that helps startups turn conversations on X into paid customers I think it could be powerful for boosting the marketing around your products!

@DataFast_ The most interesting metric isn't bot traffic. It's which questions bots are trying to answer that your product doesn't answer yet....

@DataFast_ Do you somehow block AI bot crawlers?

@DataFast_ Yes but most end up never running JS so the script don’t execute anyway

@DataFast_ Can @DataFast_ track AI brand mentions & if not is this a feature you think will work its way into the app?

@DataFast_ I can build it! Top 3 thing you want to see?

Legend! I’m signing up this week to service parts of my SEO offer for my clients. I'm probably not the best person to ask since I haven't used the software thoroughly yet, but my first thoughts are: 1. AI brand visibility 2. Search console data to see impressions clicks and CTR 3. Tracking those pages to see if their SERP position drops and they might need re-optimising Unsure if these are already baked in, but appreciate the reply, I'm loving what you're doing with DataFast.

@DataFast_ AI visibility: on it For seach console data, it's there already!

@DataFast_ Amazing thanks Marc

@DataFast_ 把爬虫行为当产品信号有价值，但也容易误判需求。AI 请求一个不存在页面，不代表真人愿意付费，最好还是和搜索词、转化路径一起看。

@DataFast_ the exact pages they crawl is the part i'd want to stare at. seeing surprises already?

@DataFast_ 250k crawls last 2 weeks 😳

@DataFast_ 250k is wild 😳

@DataFast_ I built something similar for one of my side projects see the almost the same ratio even a bit more around 400%

@DataFast_ The 404s are probably the most interesting part here. The failed requests tends to reveal expectations that nobody realized existed, especially when the same paths keep showing up. Have you also started seeing the same missing pages appear across different sites?

@DataFast_ How you even track it?

@DataFast_ why it's better than Cloudflare's native bot analytics?

@DataFast_ You don’t have to decipher a puzzle to find what’s important in your dashboard

@DataFast_ The 404 thing is actually huge. never thought about it that way, do you show which endpoints bots are hitting the most or just the ones that 404?

@DataFast_ All of them!

@DataFast_ Haha fair, i'm just starting out so my traffic is basically me refreshing. but the 404 insight is solid. speaking of - your CodeFast/ShipFast approach, is it better for solo devs than building from scratch? or does it even matter at 0 users?

@DataFast_ Keep us updated on your bench improvements ;)

@DataFast_ Heading to gym now boss

@DataFast_ Really interesting idea. How are you distinguishing between AI assistants fetching pages for answers vs bots crawling for training? Curious what signals you're using under the hood.

@DataFast_ Crazy! How do you differentiate between normal users and bots? 🤔

@DataFast_ These Bots don’t execute JavaScript usually and have a signature (verified UA and IP range)

@DataFast_ 👀 nice one! any funny discovery?

@DataFast_ The analytics tool for the A.I era. Well done @marclou 👏

@DataFast_ Nice

@DataFast_ I love how you get an idea and immediately build it. I'd love to achieve that feat

@DataFast_ The fetch pattern is a leading indicator of citation. Which pages ChatGPT pulls most often predicts what it surfaces in answers — before any citation data confirms it. That's the signal most GEO strategies are flying blind on.

@DataFast_ what is this based on? user_agent header?

@DataFast_ Yes and IP

@DataFast_ ive integrated this and was shocked haha very cool feature tho still finding ways to take info from it

@DataFast_ I use CLI a lottt too!

@DataFast_ Çoğu site sahibi bunu görüyor ama 'normal traffic' sanıyor. Asıl soru: training crawleri bloklayıp indexing'i tutmak mümkün mü? robots.txt yeterli değilse ne yapılıyor? 🧠 Günlük AI radarın: @yapayradar — takip et.

@DataFast_ 300% bot traffic is insane thats not a website thats a robot convention humans are the niche audience now lmao

@DataFast_ the 404 angle is the sharp part. if ChatGPT keeps requesting a page that doesn't exist, that's demand your users are asking the model for. that log is basically free keyword research, and the start of AEO eating SEO.

@DataFast_ What are you using for recording yourself? I'm using Loom but not sure if there is a better alternative out there.

@DataFast_ This feature is gold. I also wanted to know this things. And the best part is that 404 pages. I also built pages of users are going on but it's 404.

@DataFast_ good

@DataFast_ Seeing 404s as demand signals is clever, it turns noisy bot traffic into actionable insight. Easy to imagine how that could shape roadmap quickly

@DataFast_ super compelling. going to sign up soon

@DataFast_ Great great idea 🙏🏾