AI Crawlers Are Hitting Your Site Thousands of Times a Day — So Why Aren't You Getting the Traffic?

As of June 2026, bots generate 57.5% of HTML web traffic — machines have officially outnumbered humans on the internet for the first time. A large part of that shift is AI crawlers, which have grown fast enough over the past two years to flip the ratio. But here's the thing: if you pull up your server logs and see thousands of AI bot hits every day, most of that traffic isn't going to send you a single reader back. The gap between how much AI crawlers take and how much they return is one of the more striking numbers in web analytics right now — and it's not consistent across bots.

The crawl frequency data in this post comes from a 30-day server log study across 12 production sites by Digital Applied, using a March–April 2026 observation window. Referral ratio data is from SEOmator's GEO Data Report for Q1 2026, cross-referenced with web crawler analytics through June 2026. Industry-level traffic figures reference the HUMAN Security 2026 State of AI Traffic benchmark report.

Which bots are actually hitting your logs?

On a typical B2B or media site, GPTBot leads on raw daily request volume — a median of 4,200 page requests per day across the 12 sites in the Digital Applied study. ClaudeBot trails at around 1,800 per day, and PerplexityBot sits at roughly 980. The picture looks different on ecommerce sites, where Bytespider (operated by ByteDance) becomes the highest-volume bot by a noticeable margin — around 6,500 product-page requests per day, with unusually dense coverage of category and listing pages.

The patterns behind the numbers are also different. GPTBot revisits high-traffic pages roughly every 2.4 days on average; Bytespider cycles through commercial product pages every 1.8 days on retail properties; PerplexityBot stays mostly quiet for stretches of days and then spikes hard when a viral query triggers a batch of fetches. So when you see "AI bot traffic" in your logs, you're actually looking at at least three or four very different crawl behaviours, each tied to a different downstream use case.

What's the purpose behind all those requests?

The reason this matters is that crawl purpose determines whether you ever see a referral from it. According to web crawler analytics from May 2026, training-only purposes account for 51.8% of all AI crawler requests. Another 35.7% are classified as "mixed" — content extracted for training that also gets used for live retrieval. Search-only purposes, the kind where a live user query is driving the fetch and might result in a click back to your site, account for just 9.3% of total AI crawler activity.

The search-only slice has been slowly growing — it was around 7.5% in late 2025 and had reached roughly 10.7% by mid-2026. But it's still less than a tenth of everything AI crawlers are doing on the open web. Over half of what's in your access logs is extraction with no downstream mechanism to route a human back to you.

Does crawl volume tell you anything about referral value?

Here's where the numbers get genuinely interesting. Research published in 2026 found that GPTBot now crawls 3.6 times as many pages as Googlebot. But Googlebot operates at roughly a 5:1 crawl-to-referral ratio — five pages indexed for every eventual visitor sent back to the source. GPTBot's equivalent ratio sits at 857:1. PerplexityBot, which runs a search product with source citation, sits at around 111:1.

The Q1 2026 data showed at least one major AI training crawler operating at close to 24,000 pages crawled per referral — nearly 5,000 times worse than Googlebot. The mechanism is straightforward: search crawlers index content to power a result the user can click. Training crawlers index content to build a model that answers in natural language. When the model answers, the source isn't cited and there's no link to follow. PerplexityBot sits in the middle — it has a search product with citations, so some fraction of its crawls do eventually lead to referred clicks, which is why its ratio is dramatically better than pure training crawlers.

This asymmetry explains why aggregate AI bot traffic can be growing fast while referral traffic from AI systems stays modest. Travel and hospitality saw AI traffic volume grow 1,700% between July 2024 and February 2025, per the HUMAN Security report. The IT sector — which has the highest AI referral rate of any industry — sits at just 2.8% of visits from AI sources. Explosive growth in crawl volume, very thin growth in actual clicks.

What should you actually do with this?

The first thing worth recognising is that "AI bot traffic" isn't a single category, and a number that aggregates all of it doesn't tell you much. 50,000 GPTBot requests and 50,000 PerplexityBot requests are not the same thing. PerplexityBot traffic is connected to a system that produces citations and can drive readers back; GPTBot traffic is primarily connected to a training pipeline that doesn't.

One thing the data does highlight is that blocking AI crawlers wholesale — which became easier after several platforms added one-click bot-blocking features in 2025 — is a more nuanced decision than it first appears. If you block all AI training crawlers, you lose training influence but retain bandwidth. If you block selectively, PerplexityBot at 111:1 is a very different calculation from a training crawler at 24,000:1. The crawl-to-referral ratio is a useful signal for making that call.

For the longer game, training crawls matter even without a direct referral. A model that ingested your content during training has learned something from it — that shapes how AI systems describe your product, recommend your category, or cite your research later. That's what's sometimes called the GEO (generative engine optimisation) frame. The practical upshot is that both goals — referral traffic and training influence — require the same kind of content: static, well-structured, readable by an HTTP fetcher that doesn't run JavaScript. If your page renders blank without JS, you get neither the referral value from PerplexityBot nor the training value from GPTBot. That's the one thing every AI crawler has in common.