Which AI Bots Are Actually Giving Back? The Crawl-to-Referral Gap
ClaudeBot crawled 23,951 pages for every single referral visit it sent back in Q1 2026. PerplexityBot's ratio was 111:1. GPTBot's 1,276:1. Here's how to read these numbers — and what they should mean for your robots.txt policy.
ClaudeBot averaged 23,951 page crawls for every single visit its parent platform returned to publishers in Q1 2026. For PerplexityBot, that ratio was 111:1. GPTBot sat at 1,276:1. DuckDuckGo, for comparison, was 1.5:1. These numbers come from CDN-level radar logs tracked across the first quarter of 2026, and they describe something genuinely useful: the exchange rate between what each AI bot takes from your site and what its parent platform sends back.
Most bot management conversations focus on volume — how many requests, how much crawl budget. The crawl-to-referral ratio reframes the question as one of return on access. Granting a crawler access costs you server resources and crawl budget. The question is what you get back.
Where does this data come from?
The crawl-to-referral ratio figures come from a Q1 2026 GEO analysis published by SEOmator, using CDN-level radar data from January through March 2026. The methodology divides total pages crawled by a bot against the referral sessions its parent platform sent to publishers over the same window. Crawler volume data (daily page requests) comes from a separate 48-day server log study across twelve sites, published by WISLR. These two sources measure different things but together build the full picture: volume tells you cost, ratio tells you return.
What does the ratio actually mean per bot?
The spread isn't random noise — it maps almost exactly onto product strategy.
PerplexityBot returns the most traffic relative to what it crawls. At 111:1, it has the lowest ratio among AI-specific crawlers. Perplexity operates a search product that prominently cites sources with working links; the citation model is central to how it positions itself. The return rate reflects that architecture.
GPTBot at 1,276:1 sits in the middle. ChatGPT does return some referral traffic — particularly from its web-browsing mode where it includes cited links — but the bulk of GPTBot's crawling is for training data rather than real-time retrieval. The ratio is bounded by how often ChatGPT's browsing mode is actually used versus how much the underlying models consume for training.
ClaudeBot at 23,951:1 is in a structurally different position. ClaudeBot's parent company currently operates no consumer search product that links back to publishers. ClaudeBot indexes content to improve model capabilities, not to power a retrieval system that generates referrals. So the ratio is not a gap to close — it's the intended operating mode.
DuckDuckGo at 1.5:1 benchmarks what a traditional search engine looks like by comparison: roughly one referral per crawl. That baseline matters because it shows what "a good exchange rate" can look like when the product architecture aligns.
How much are these crawlers requesting per day?
The 48-day study found GPTBot averaging 4,200 page requests per site per day — the highest among AI bots. ClaudeBot averaged 1,800. PerplexityBot came in at 980.
At GPTBot's ratio, those 4,200 daily crawls translate to roughly 3.3 referral visits per day per site — about 100 referral visits per month per site. For some categories of site that's meaningful; for others it's barely perceptible. What makes the ratio framing useful is that it converts raw volume into something measurable against the actual outcome you care about.
The server resource cost of 4,200 daily GPTBot requests depends entirely on how your infrastructure handles them. Static pages cached at the edge cost almost nothing. Dynamically generated pages or pages that hit a database on each request cost considerably more. If you're seeing AI crawlers compete with your normal crawl budget or push CDN costs upward, server logs will tell you — GA4 won't.
What does blocking actually change?
A Q1 2026 analysis of robots.txt configurations across a large CDN network found GPTBot in DISALLOW rules at 5.52% of sites — the highest block rate among AI crawlers. ClaudeBot was blocked at 4.88%. PerplexityBot was blocked far less frequently, consistent with its higher referral return rate making it a more attractive crawler to keep.
The catch is that a robots.txt DISALLOW controls future crawls only. Content already indexed before the block was added remains in whatever training sets or retrieval indexes already ingested it. And for bots that primarily crawl for training rather than retrieval, blocking removes future content from training data but doesn't retroactively affect how the model describes your existing products or brand.
There's also a subtler effect to account for: AI crawlers can influence how AI assistants discuss your brand or content even without returning referral traffic. A model trained on your content may describe your product accurately; a model that never saw your content may not. Whether that effect is worth the crawl overhead is a separate question from the referral return rate — but it's a reason not to treat blocking as a free action.
What should you actually change?
Use the ratio to make per-bot decisions. "Should I allow AI crawlers?" is not a useful question when PerplexityBot at 111:1 and ClaudeBot at 23,951:1 are presenting fundamentally different propositions. A blanket policy in either direction ignores the difference. The ratio gives you the variable to optimize against.
Separate training crawlers from retrieval crawlers in your logs. Most log analysis setups lump all AI bots together. Splitting them by function — retrieval agents like ChatGPT-User and Perplexity-User (real-time browsing) versus training crawlers like GPTBot and ClaudeBot (batch indexing) — lets you see which category is generating actual referral sessions versus which is consuming crawl budget for other purposes.
Recheck your policy at least twice a year. GPTBot's referral rate has room to improve as the web-browsing product evolves. PerplexityBot's ratio may shift as it scales. A robots.txt decision made on Q1 2026 data may be the wrong call by Q3. Set a review cadence that matches the pace the AI product landscape is actually moving at.
The crawl-to-referral ratio won't tell you everything — it doesn't capture brand visibility effects, citation quality, or how model training affects long-run referral rates. But it's a cleaner variable for bot access decisions than crawl volume alone, because it encodes both the cost and the return in a single number.