Which AI bots are actually crawling your site — and how many of them ever send you a visitor?
Training crawlers now account for over half of all AI bot requests, and they send back almost nothing. Here is what your server logs reveal that your analytics tools won't.
Over half the AI crawler requests hitting your site right now aren't looking for content to surface to users — they're harvesting it for model training. That share was 41.1% at the start of 2026. By May it had climbed to 51.8%, with "mixed-purpose" crawling (training plus retrieval) adding another 35.7%. Pure search indexing accounts for just 9.3% of all AI bot requests. So the question worth asking isn't "is my site being crawled by AI?" It almost certainly is. The real question is: what are those crawlers actually doing with your content — and will any of them ever send you a visitor?
Where this data comes from
The numbers here come from three sources: edge-network AI Insights telemetry tracking verified bot requests across a large portion of the global web; a July 2026 GEO data report from SEOmator that measured crawl-to-refer ratios across thousands of publisher sites; and a 30-day server log study from DigitalApplied covering agentic crawler behavior across a sample of mid-traffic sites. All three independently point to the same split: most AI crawler volume is training-oriented, not search-oriented.
Which bots are actually showing up in your logs?
As of May 2026, the verified bot landscape for AI-adjacent traffic breaks down roughly like this:
- Googlebot: still the single largest at 27.26% of AI-adjacent bot requests
- GPTBot: 11.48%
- Bytespider (ByteDance): 10.25%
- ClaudeBot: 9.73%
The AI-native bots combined now account for roughly 26.7% of all verified bot traffic. Worth noting: these numbers are volatile month to month. In April 2026, ClaudeBot was ahead of GPTBot (11.69% vs 9.84%); by June, ClaudeBot had nearly doubled to 19.8% while GPTBot eased to 9.4%. Monthly snapshots tell you the direction, not the final destination.
Does any of this crawl traffic actually turn into visitors?
This is the number that changes how you should think about AI bots. The crawl-to-refer ratio measures how many pages a bot crawls for every referral visit it generates. A ratio of 100:1 means the bot consumed 100 of your pages before its platform directed a single user to your site.
The gap between platforms is staggering. GPTBot crawls 848 pages per referral sent. ClaudeBot crawls 4,580. Compare that to Googlebot at 5:1 — five pages crawled per click. PerplexityBot sits at 186:1, and Microsoft Copilot at 33:1.
What does that mean practically? If you're optimising for referral traffic, the training-heavy crawlers aren't your best bets. PerplexityBot and Copilot are delivering much better referral economics right now — lower crawl volume, more clicks per page consumed. The high-ratio crawlers (GPTBot, ClaudeBot) are essentially mining your content without a strong referral loop. Whether your content being in those training datasets generates indirect value — brand familiarity, your terminology appearing in model responses, your domain getting cited in answers — is hard to attribute but increasingly worth thinking about.
Do training bots and search bots behave differently on the page?
Yes — and server logs make this obvious. From the 30-day log analysis:
GPTBot crawls breadth-first, averaging around 4,200 requests per site per day at a mean path depth of 3.8. It gravitates to /blog/, /docs/, and /about/ — long-form text it can distil. Revisit cadence is aggressive: roughly every 2.4 days, and 47% faster on pages with a fresh Last-Modified header.
ClaudeBot crawls depth-first (mean path depth 5.2 vs GPTBot's 3.8), averaging 1,800 requests per site per day. It prefers /docs/ and /api/ paths — structured, referenceable content rather than blog posts. Revisit cadence is slower, roughly every 6.8 days.
PerplexityBot is burst-driven rather than scheduled. Baseline traffic is quiet, but when a user query references your domain, the bot can spike to 240 requests per minute on a publisher site. It favours text-heavy, citable content — /blog/, /docs/, and commercial comparison pages.
These patterns map directly to each platform's end product. GPTBot wants long-form text it can summarise; ClaudeBot wants structured reference material to draw answers from; PerplexityBot wants what users are actively searching and comparing. Which of those end products is most likely to feature your content prominently? That's the bot worth optimising for first.
The emerging third category: agentic crawlers
Beyond the training/search split, there's a third category worth watching: autonomous bots that browse the web as part of completing a user task, rather than bulk-crawling for data. Traffic from these systems grew an estimated 7,851% year-over-year through 2025. That's not a typo.
They don't match traditional crawler user-agent strings cleanly, and they don't behave like either training or search bots — they follow multi-step flows, interact with JavaScript-heavy pages, and often target functionality rather than content. For most sites, agentic traffic is still a rounding error in the logs. But the trajectory matters: today's niche category is yesterday's novelty and tomorrow's mainstream. If you're seeing unusual short request sequences targeting your checkout or account pages, that may be agentic crawl traffic rather than traditional scraping.
What should site owners actually do with this?
The headline number — AI crawler traffic up 187% year-over-year through 2025 — tends to dominate the conversation. But it flattens a distinction that has real strategic implications. Training crawlers and search crawlers have different incentive structures, different crawl signatures, and very different referral economics.
If referral traffic is the goal, PerplexityBot and Copilot are better bets right now. They crawl less aggressively, generate more referrals per page consumed, and their platforms are building discovery surfaces that actively cite sources. If you've been blanket-blocking AI crawlers in robots.txt because the bot traffic volume annoys you, you might be blocking the ones most likely to actually send you users.
If training inclusion matters, that's a separate calculation. The major training crawlers have high volumes and low referral rates by design — they're building models, not indexes. Whether your content in those datasets creates indirect value is hard to measure but increasingly a strategic question for developer-facing and B2B products, where model familiarity with your brand's terminology and API surface area can drive inbound demand.
One broader trend worth watching: block rates for crawler requests went from under 10% to over 33% of requests in the past year. A third of AI crawler requests are now getting bounced. That changes what the aggregate data means — crawler statistics increasingly reflect who's allowing which bots, not just who's sending them. If you've blanket-blocked all AI user agents and you're wondering why your brand isn't surfacing in AI search results, the robots.txt file is probably the first place to look.
The only reliable way to see any of this is in your server logs. Standard analytics dashboards don't record bot visits. Filter your access logs by the known AI user-agent strings — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot — and you'll find a traffic category that's been invisible to most site owners until now, and is only going to get bigger.