1,917 Pages Crawled, One Referral Back: What's Your AI Bot Traffic Really Worth?
AI crawlers now account for nearly a third of all web traffic — and the ratio of pages crawled to visits sent back is staggering. Here's what the data shows.
Something odd turns up when you dig into server logs on almost any mid-sized website right now. Requests from well-known AI bots are stacking up — sometimes hundreds per hour — while referral sessions from AI-powered products stay basically flat. The numbers don't add up. Or rather, they do add up, just not in your favour.
By Q3 2025, AI bots were accounting for 31.5% of web traffic at the median — roughly 46 AI requests for every 100 human visits, according to Senthor's State of AI Bots research. By June 2026, automated traffic had pushed past humans entirely, generating 57.5% of all HTML page requests across major content delivery networks. That's a pretty dramatic flip in under a year.
So what are all those bots actually doing — and is any of it finding its way back to your site?
Are the bots in your logs training crawlers or live user queries?
This is the question most site owners aren't asking, and it matters more than almost anything else about how you respond to AI bot traffic.
Security research firm Fastly analysed 6.5 trillion monthly requests across their network from April to July 2025 and found that roughly 79% of AI bot traffic came from training crawlers — bots bulk-harvesting content to build or update AI models. The other 21% were "fetcher bots", which activate when a real user fires a query into an AI assistant and the assistant needs to pull live content.
These two categories interact with your content completely differently. A training crawler follows sitemaps, processes text in bulk, and doesn't execute JavaScript. It's reading your page the same way you'd copy text out of a Word document. A fetcher bot is a different animal — there's a live user query behind it, it needs structured, readable content, and it's probably in a hurry.
Who was generating all that training crawler volume? In the Fastly data, Meta's bots (operating under Meta-ExternalAgent) held 52% of training crawl traffic. Google's training crawler (Google-Extended) came in at 23%, and the bots operated by the company behind ChatGPT at 20%. On the fetcher side, the same company's real-time browsing bots accounted for 98% of all fetcher traffic — a near-monopoly on live-query access.
If your site renders most content via client-side JavaScript, a training crawler sees an empty shell. That's 79% of your AI traffic reading nothing.
Who are the biggest movers in your logs — and does rank even matter?
Looking at verified bot traffic in early 2026, the market share breakdown puts Googlebot at 38.7%, followed by GPTBot at 12.8%, Meta-ExternalAgent at 11.6%, and ClaudeBot at 11.4%. These are the names you'd expect to see dominating your access logs.
But what's the actual trend? GPTBot grew 305% in raw request volume between May 2024 and May 2025. ClaudeBot surged roughly 66% in a single month in mid-2026. By July 2026, ClaudeBot had climbed to 16.3% of AI crawler traffic and GPTBot had dropped back to 9.7%. The companies behind these bots are adjusting their crawl rates fast — often without announcing anything.
Any single snapshot is outdated within weeks. The more useful thing to track isn't who's number one this month, but whether any of this volume is converting into actual visits.
Are AI crawlers actually sending you any traffic at all?
Here's where things get uncomfortable. The crawl-to-refer ratio measures how many of your pages a bot reads per referral session it eventually sends back to your site. A search crawler with a 5:1 ratio is efficient — it reads five pages to understand your content, then sends you a visitor. An AI bot with a 2,000:1 ratio is something different entirely.
As of July 2026, Googlebot sits at roughly 5.2:1. DuckDuckGo's DuckAssistBot is even tighter, at about 1.5:1. GPTBot's ratio has improved significantly — down from 1,104:1 in July 2025 to 251:1 a year later — which is real movement, even if it's still 50x worse than Googlebot. ClaudeBot sits at 1,917:1 over the same period.
For every session ClaudeBot sends back to your site, it's reading roughly 1,917 of your pages first.
Should this send you straight to your robots.txt to start blocking? The data isn't encouraging on that front. Research published in late 2025 found that sites blocking AI crawlers via robots.txt actually saw a 23.1% decline in total monthly visits and a 13.9% drop in human-only browsing. GPTBot is now the most-blocked AI crawler on the web, appearing in 5.52% of DISALLOW rules as of Q1 2026, with 25% of the top 1,000 websites blocking it outright. Whether there's a causal relationship or just correlation with other factors, a blanket block carries real risk.
The ratio will likely improve over time as AI products expand their citation behaviour. GPTBot going from 1,104:1 to 251:1 in twelve months is meaningful — the referral economics are slowly shifting, even if they haven't reached search-engine territory yet.
So what should you actually do with this information?
Given the training/fetcher split, a few things are worth checking right now:
Is your content actually visible without JavaScript? Pull your key landing pages with a bare curl request and look at what comes back. If the response is mostly empty divs, the 79% majority of AI training crawlers is getting nothing from your site. Serving pre-rendered HTML to known bot user-agents changes this immediately.
Are AI bots concentrating on specific sections? Bots tend to cluster on content-heavy paths — blog archives, documentation, product descriptions. If most of your AI bot hits are on URLs you wouldn't prioritise for normal SEO, that's a signal about where your value in AI training sets actually lives.
Is a blanket block the right call for you? For most sites, probably not — especially if you're betting on future referral traffic from AI products whose citation behaviour is gradually improving. The more surgical approach is to allow fetcher bots access while applying rate limits or crawl-delay rules to training crawlers.
The volume of AI bot traffic isn't going down. Market share is shuffling every month, the crawl-to-refer ratios are edging lower, and the training vs. fetcher distinction is getting clearer in server logs. What's most certain is that a site serving blank JavaScript shells to crawlers is leaving a lot on the table — whether that eventually means referral traffic, AI citations, or something that doesn't quite exist yet.