What Are Those Thousands of Daily AI Bot Hits Actually For?

Nearly half of every AI bot hit your server receives right now is going into a training pipeline that has nothing to do with sending you readers. Training crawlers now account for 44.5% of all AI crawler traffic as of July 2026, up from 35.7% a year earlier. If you have not revisited your robots.txt since that split happened, you are almost certainly making access decisions with an outdated picture of who is actually crawling your site.

So who is crawling it? And what does any of it have to do with your actual traffic numbers?

Where the data comes from

The figures here draw from three public sources published between May and August 2026: a 30-day log study across twelve production sites spanning B2B SaaS, e-commerce, and publishing (Digital Applied), a comparative crawl analysis across 66 billion requests (Colonel Server), and per-crawler volume data from Presenc AI's half-year tracker. Where sources agree, the figures are solid. Where they diverge, the range is flagged.

What is actually inside that 22% of bot traffic?

The headline stat — AI bots now make up 22% of all bot traffic — bundles three fundamentally different types of request into one number. Treating them as a single category is where most site owners go wrong.

Training crawlers (GPTBot, ClaudeBot, CCBot) archive content for model training datasets. They have no direct connection to search citations. Crawling your site for training data does not make you more likely to appear when a user asks an AI assistant a question. Search and retrieval bots — OAI-SearchBot, PerplexityBot, and their equivalents — index pages to answer live user queries. Agent bots are triggered by actual user sessions: someone asked an AI assistant something in real time, it went to fetch data, and that request just landed on your server.

The practical implication: blocking GPTBot removes your content from training data collection. It does nothing to your chances of being cited in a live AI answer. Those are separate user-agents with separate purposes, and your robots.txt rules should reflect that.

Which bots are hitting your server the hardest?

If you want to understand where your crawl load is actually coming from, per-bot crawl frequency tells a cleaner story than overall percentages.

GPTBot is the most aggressive by a wide margin — a median of 4,200 page hits per site per day across the 30-day study. ClaudeBot comes in at 1,800 per day, PerplexityBot at 980, and Google-Extended at 540. GPTBot also tends to revisit your highest-traffic pages every 2.4 days on average; the others run less predictable schedules.

One significant shift happened in March 2026: GPTBot and ClaudeBot both started consuming XML sitemaps for the first time. If your AI bot traffic spiked unexpectedly this spring, that is a likely contributor — and it also means these crawlers now have a more structured view of your content architecture than they did twelve months ago.

So what is the actual payoff in referrals?

Here is where the numbers get uncomfortable. All of that crawling is not translating into proportional traffic back to your site. AI platforms collectively account for around 0.32% of total website traffic as of mid-2026, up from 0.24% in 2025 and 0.02% in 2024. That 16x growth over two years sounds substantial until you put it next to the 87.6% of search referrals that still come from Google.

The crawl-to-referral ratio tells a more specific story.

For the dominant AI search platform, the ratio stood at approximately 1,104:1 in July 2025 — for every visit it sent back to the web, it had crawled over a thousand pages first. By July 2026, that had improved to 251:1. Still deeply asymmetric, but the direction of travel is right. As AI assistants get better at surfacing cited sources, the gap between crawling and referrals should keep narrowing.

Among AI platforms that do send referrals, ChatGPT accounts for somewhere between 74% and 92% of the AI referral pool depending on methodology. Perplexity was a more significant contributor earlier in 2025 but its outbound referral volume dropped when it pivoted toward keeping users inside its own browser and agent interface.

How are sites responding — and what are they getting wrong?

About 84% of websites have no AI crawler policy in their robots.txt at all. Among those that do, 25% of the top 1,000 sites block GPTBot outright. That might be the right call for keeping content out of training datasets — but many of those sites have not realised that the citation-focused search bot is a completely separate user-agent that would not be affected by a GPTBot-only rule.

Roughly 27% of B2B sites are accidentally blocking major AI crawlers through CDN-level rules they did not configure intentionally, often without anyone on the team knowing. Meanwhile, 79% of top news publishers have explicitly allowed citation bots through while blocking training crawlers — they drew the distinction deliberately. Most other sectors have not.

There is also a compliance gap worth noting: 39.5% of GPTBot bans in robots.txt appear to be inconsistently enforced. If you believe you have blocked it based on your robots.txt configuration alone, your server logs are worth checking.

What should you actually do about this?

If you have been treating all AI crawlers as a single bucket, that mental model needs updating. The question is not whether to allow AI bots — it is which types, and for what purpose.

If you want to be cited in live AI answers, the bots that matter are the search and retrieval crawlers. Blocking training crawlers has zero effect on whether you show up in those answers. Your robots.txt needs to reflect that distinction rather than making a blanket call that accidentally covers both categories.

If your site runs behind a CDN, audit those rules separately. A blanket CDN-level block hits training and citation bots equally, and it is almost certainly not a trade-off you thought through deliberately when you first set it up.

The referral numbers are small today — 0.32% of all traffic — but the trajectory matters. We went from 0.02% to 0.32% in two years. The crawl-to-referral ratio is falling. The bot categories are splitting into more purpose-specific crawlers. The practical starting point: pull your server logs, count hits by user-agent string, and figure out which bots you are actually dealing with before your robots.txt makes the decision for you.