Who's Really Crawling Your Site — and Should You Care Which Bot It Is?

Twenty-two percent of all verified bot traffic on the web is now AI-driven — and that number has roughly doubled in two years. If you're checking your server logs and noticing strings like GPTBot, ClaudeBot, or PerplexityBot appearing dozens or hundreds of times a day, you're not imagining it: the volumes are real and have been climbing fast. But here's a question most site owners haven't really sat with yet — do all these crawlers behave the same way, and does it actually matter which one reaches your site?

How we got these numbers

The crawler market-share data here comes from a large-scale monthly tracking report that monitors AI bot activity across millions of sites. Crawler-purpose classification — training versus real-time retrieval versus agentic — is drawn from a longitudinal study covering 2022 to 2026. Robots.txt blocking patterns come from an analysis of roughly one million domain files conducted in Q1 2026.

Which crawlers are winning — and which are losing ground?

A year ago, GPTBot held the larger share among training-class crawlers: 13.15% versus ClaudeBot's 11.23%. By July 2026 those positions had reversed decisively. ClaudeBot sits at 16.28%, GPTBot at 9.74%. That's not a gradual drift — it's a significant structural shift in who's touching your content most often.

What's behind it? The organisation running ClaudeBot appears to have substantially expanded its crawl footprint from late 2025 onwards. Sites that previously saw it weekly started seeing it daily. GPTBot's decline is likely deliberate — its operator has been building out a separate real-time retrieval agent rather than running large batch training sweeps, which changes what the training crawler actually needs to do.

The broader lesson here is that the leaderboard isn't stable. If you configured your robots.txt rules in 2024 based on which crawler was dominant at the time, that picture has already changed. Two of the top-four AI crawlers by volume have switched positions since then.

Training crawl or real-time agent — does the distinction matter?

Not long ago, the story was simple: AI bots crawled your site to build training datasets. In 2026, the picture is considerably more complicated. Pure training crawlers account for 44.8% of AI bot requests — a plurality, but no longer a majority. Mixed-purpose crawlers (indexing for both training datasets and live retrieval) account for a nearly equal 42.7%. Search-focused indexing adds another 9.3%, and user-action agents — bots completing a real-time task on behalf of an actual user — are at 2.6%.

That final category is the one that should get your attention. At 2.6% it looks small, but agentic crawl traffic grew by more than 7,000% across 2025. These aren't batch training sweeps from a data centre; they're real users asking AI assistants to look something up, and the assistant is visiting your site as part of that task. If your site returns blank content — because it relies on client-side JavaScript rendering — the agent sees nothing, and the user gets told your page wasn't accessible.

Training crawlers are far more forgiving by comparison. They capture whatever static text is available and move on quickly. No one has a worse experience if a training sweep misses some content. The retrieval and agent tiers are where the stakes are actually high.

Are sites blocking the right crawlers?

Here's something that jumps out from the blocking data: GPTBot is still the most-blocked AI crawler in robots.txt files, appearing in DISALLOW rules on 5.52% of domains sampled. But its traffic share has now fallen to under 10%. ClaudeBot — currently the larger crawler by volume — gets blocked on just 4.88% of domains. The blocking policies and the actual traffic mix are increasingly out of sync.

Why the mismatch? Most blocking decisions were made in 2023 and 2024, when GPTBot was the clear front-runner and there was a significant public debate about web scraping for AI training. Those rules went into robots.txt files and largely stayed there. Sites that blocked GPTBot on principle are now inadvertently running an asymmetric policy — the smaller crawler is blocked while the larger one has free access.

It's also worth asking whether robots.txt blocking achieves what site owners expect. The major AI labs have publicly committed to honouring these rules for their training crawlers, and the compliance record is generally good. Retrieval and agent bots are more variable — some check robots.txt before every fetch, others use cached policy data that may be weeks old.

What does this mean for you?

If you're running a JavaScript-heavy site, the retrieval and agent bots are the ones creating an actual problem right now. These are the crawlers that, when they fail to get useful content, actively degrade what a real user experiences — their AI assistant returns a thin or missing answer about your content. Pre-rendering fully-rendered HTML to detected AI bot user-agents addresses this directly.

Your robots.txt is probably stale. Pull 30 days of server logs, look at which AI user-agent strings are actually hitting your site, and compare them against a current crawler list. There's a reasonable chance you're blocking last year's leading crawler while leaving this year's higher-volume one unconfigured.

And if you haven't considered the agent tier yet — the 2.6% that's been growing at orders of magnitude — now is worth thinking about. These bots behave like real users but can't execute JavaScript or handle client-side redirects. Structured, server-rendered content is what they need to successfully represent your site when someone asks an AI assistant about it.