Who's Really Crawling Your Site — and Are Any of Them Sending You Visitors?

Twenty-three thousand pages crawled for every visitor sent back. That's the crawl-to-referral ratio for one of the most active AI training crawlers hitting the web right now — it processed 23,951 of your pages on average before its parent platform directed a single user your way. AI crawlers are consuming a growing slice of the web's bandwidth, but are they actually worth anything to your site? The honest answer depends entirely on which bot we're talking about, and most analytics setups aren't making that distinction.

Where does this data come from?

The figures below pull from three main sources: websearchapi.ai's monthly crawler report for June 2026 market share data, a 30-day study of 12 production sites published by DigitalApplied in April 2026 for crawl frequency numbers, and seomator.com's GEO data report for crawl-to-referral ratios. All three use server log analysis rather than self-reported vendor data, so they capture actual HTTP requests including bots that don't always announce themselves cleanly.

Which bots are actually showing up right now?

The June 2026 bot-mix was unusually volatile, even by recent standards. Googlebot still leads at 24.9% of AI-adjacent bot HTTP requests — but that's a new all-time low, down from 30.28% in April. The big story from June was ClaudeBot surging +66% month-over-month to become the #2 AI crawler at 20.0%, jumping over both Meta-ExternalAgent (16.7%) and GPTBot (12.0%) in a single month. Bytespider, which nearly doubled its share in May to hit 10.25%, reversed sharply to 7.3%.

Why so volatile? Training crawls are bursty by nature. When an AI company kicks off a large-scale crawl to update model weights, that bot's share spikes hard for the duration of the run and then drops back. ClaudeBot's June surge almost certainly reflects a scheduled training run more than any organic shift in end-user adoption. If you check back in August, the numbers will probably look different again. That's not a reason to ignore the data — it's a reason to monitor continuously rather than treating any single month as a settled baseline.

What are these bots actually doing when they show up?

Here's the distinction that most site owners miss: not all AI bots are training crawlers.

When you break down AI bot traffic by intent, real-time user-triggered agents make up 44.3% of hits. These are bots fetching your page the moment a real person asks an AI assistant a question that references your domain — someone is literally asking about your content right now. Training crawlers (the bulk-download, model-update kind) are 29.6%. Retrieval bots that ground AI answers in real-time indexed content add another 15.1%.

That 44.3% real-time figure matters a lot for how you think about AI traffic. A real-time agent visit correlates to conversions in a way that a training crawl — which might not ship as a model update for months — simply doesn't.

There's also a practical implication for robots.txt. You can block training crawlers without losing AI citations, because they use separate user agents. The bot that handles bulk training data collection and the bot that handles real-time search indexing are different agents with different names. Most site owners who added a training bot disallow two years ago don't realise they can still appear in AI search answers by leaving the search and retrieval agents unblocked. Worth checking.

Are any of them actually sending you traffic back?

This is where the math gets rough. The crawl-to-referral ratios for AI bots span four orders of magnitude.

Microsoft Copilot, which surfaces sources prominently in its answers, returns about 30 referral visits per 1,000 pages crawled. PerplexityBot sits at 9 per 1,000. GPTBot sends 0.8 per 1,000 — meaning it crawls 1,276 of your pages per referral visit. And ClaudeBot sits at approximately 0.04 referral visits per 1,000 crawls, or about one referral for every 23,951 pages indexed. Googlebot, for context, has held at around 200 per 1,000 (a 5:1 ratio) for a decade.

Does this mean training crawlers are worthless? Not necessarily. Being in an AI model's knowledge base has value that doesn't show up in your referral analytics — if an AI assistant confidently recommends your product because it ingested your documentation, you may never see a training bot referral in your analytics but still win the conversion. What the data does tell you is that PerplexityBot referrals deserve more weight than their crawl volume suggests, and the training crawlers deserve more scrutiny on the robots.txt side given what they're costing you in crawl budget.

How hard are they hitting you on a daily basis?

On raw request volume, GPTBot leads by a significant margin. A 30-day study across 12 production sites found it averaging 4,200 requests per site per day — nearly 2.4x ClaudeBot's 1,800 daily hits, and more than 4x PerplexityBot's 980.

GPTBot also crawls breadth-first, heavily favouring content-dense paths: /blog/, /docs/, /about/, and pricing pages. That means it's consuming crawl budget fast on exactly the pages you care most about. ClaudeBot's pattern is different — it's more event-driven than scheduled, tending to spike when your domain comes up in user conversations rather than running a predictable background sweep.

So what should you actually do with this?

Three things worth doing if you haven't already.

First, audit your robots.txt against the current list of AI agent user agents. The landscape has expanded significantly in the last 12 months — new training agents, new retrieval agents, new real-time user agents. A robots.txt written in 2024 almost certainly isn't making the training-vs-search distinction you'd want today. The goal isn't to block all AI bots — it's to block training crawls you'd rather not fund while leaving search and real-time agents accessible.

Second, if your site is JavaScript-heavy, understand that all of these bots are fetching your HTML statically. None of the major AI crawlers execute JavaScript as of mid-2026. If your product pages, docs, or blog posts rely on client-side rendering, they're effectively blank to every bot on this list. Pre-rendered HTML isn't optional for AI visibility — it's the baseline requirement before any of the optimisation work matters.

Third, pay attention to PerplexityBot and Copilot's crawl agent specifically. They have the best crawl-to-referral ratios of any AI indexer right now, and they're the ones most likely to surface your content as a cited source rather than just ingesting it into a model. If you're going to prioritise any AI crawler access in your caching and content structure, prioritise the bots that actually send visitors back.

The bot-mix will look different in September. But the split between training crawlers, retrieval bots, and real-time agents isn't a temporary distinction — it's how the AI ecosystem is structured, and if your analytics isn't making that split, you're making content and infrastructure decisions on incomplete data.