Why Are AI Crawlers Hitting Your Site 4,500 Times for Every Visitor They Send Back?

ClaudeBot crawled roughly 4,580 pages on publisher sites for every referral visit it sent back in Q1–Q2 2026. Google's traditional search crawler ran at about 5:1 over the same period. That's a 900x gap — and it's either an argument for selective blocking or a reason to rethink your robots.txt strategy from scratch, depending on which side of the equation you're on.

Where this data comes from

Crawl-to-referral ratios are now tracked by CDN providers that see both the inbound bot requests and the outbound referral headers. The numerator is total pages requested by a given user-agent string; the denominator is referral sessions attributed to the same company's platforms. The ratios in this post come from Q1–Q2 2026 public datasets cross-referenced with independent publisher research. Sources are listed below.

Is the 900x ratio gap actually about bad behaviour?

Not really, no. PerplexityBot sat at roughly 186 pages crawled per referral sent in mid-2026. GPTBot's operator landed around 848:1. ClaudeBot came in at approximately 4,580:1. And Google's standard search crawler? About 5:1.

The gap isn't about malice. It's almost entirely about what the bot is doing. Major AI companies typically run multiple distinct crawlers: one for training data collection, one for real-time retrieval that powers their search and answer products, and one for agent-mode browsing — where an actual user has asked the AI to visit a specific URL right now. Training crawlers have no mechanism that puts a human in front of your site afterward. There's no search results page, no AI answer citation box, no recommendation panel — just bulk content ingestion.

When you average all crawl types together for a single company, you get their blended ratio. ClaudeBot's 4,580:1 reflects a period when a training run dominated its request mix. If you isolated only retrieval and agent-mode bots from the same operator, the ratio would be dramatically lower.

So what is the AI traffic actually doing on your site?

In a June 2026 server log sample, the breakdown by crawler purpose looked like this: agent-mode traffic — where a real user is actively asking an AI assistant to browse a URL — came in at around 44% of request volume. Pure training crawls accounted for about 30%. Retrieval crawls that feed AI answer engines came in at 15%, and standard search indexing at 11%.

Here's the number that probably doesn't match your intuition: real-time fetches triggered by an actual person amount to roughly 2.6% of total AI crawler requests. The remaining 97.4% is background collection — training runs, index-building, retrieval cache warming — none of which sends a user to your site in the moment the crawl happens. The value, if any, is deferred: a training crawl might contribute to an AI citation six months later. But that's a very different proposition from the 5:1 ratio you get from a traditional search crawler.

The market shifted dramatically in June 2026

ClaudeBot went from outside the top three to the #2 AI crawler in a single month, jumping +66% to reach 20.0% of tracked AI crawler traffic — overtaking GPTBot and Meta-ExternalAgent in what independent trackers described as the largest single-crawler month-on-month share gain on record. At the same time, Googlebot fell below 25% share for the first time ever, landing at 24.9%. Bytespider, which had been surging, reversed sharply back to 7.3%.

Moves like this are exactly why static robots.txt lists fall behind. When the #2 crawler doubles its traffic share in 30 days, you need rules that describe what you want to allow by purpose, not just by user-agent name.

How are publishers actually handling this?

Most blocking strategies are blunt. Around 25% of the top 1,000 websites block GPTBot as of early 2026, up from just 5% in early 2023. For news publishers, 79% use robots.txt to block training bots. But the blocking patterns are mostly all-or-nothing: either disallow the well-known training bot user-agent strings, or allow everything.

What's still rare is purpose-based access control — blocking training crawlers while explicitly keeping retrieval and agent-mode bots open. This matters because the same company often runs a separate user-agent for each purpose. If your goal is to exclude your content from training datasets while still being cited in AI search results, you need separate rules for each bot type from the same operator, and you need those rules to stay current.

What should you take away from this?

First, do you know your own crawl-to-refer ratio per bot? Your server access logs plus referral analytics will tell you. If a crawler is hitting you 80,000 times a month and sending you 40 sessions, that's worth knowing before you decide to allow or block it.

Second, have you mapped the purpose behind each user-agent string for the bots you care about? Most AI companies publish this documentation — training bots, retrieval bots, and agent-mode bots get different names. Knowing which is which changes how you should think about access policy.

Third, how often do you revisit your crawler rules? The June 2026 ClaudeBot surge is a reminder that the bot landscape can shift dramatically in a single month. Access rules based on a point-in-time user-agent list will drift out of alignment faster than you might expect.

The 900x difference in crawl-to-refer ratios isn't noise — it's structural, and it tells you something specific about the value you're getting (or not getting) from each bot. Treating them all the same is a decision by default, not design.