Why Do AI Crawlers Hit Dead Pages Four Times More Often Than Googlebot?

Around a third of every request that GPTBot sends out lands on a page that doesn't exist. The same is true for ClaudeBot. Analysis of request logs from a major web hosting network in late 2024 found that GPTBot returned a 404 error 34.82% of the time, ClaudeBot at 34.16%. Googlebot, by comparison, hit dead pages just 8.22% of the time. That's a four-to-one gap — and it raises a question that most AI crawler coverage skips entirely: if a third of AI crawler requests are wasted on deleted or moved pages, what does that actually say about the value of the AI traffic showing up in your server logs?

Where this data comes from

The 404 rate figures come from Vercel's analysis of their hosting network logs published in December 2024, covering billions of requests across hundreds of thousands of web properties. Crawler growth and market share data comes from CDN-level verified bot traffic reports covering May 2024 to July 2025. The llms.txt adoption figures come from a June 2026 study of the world's top 1,000 websites.

Why are AI crawlers hitting so many dead pages?

The 404 gap between AI training crawlers and Googlebot isn't random — it reflects a fundamental difference in how they maintain their URL lists. Googlebot runs a continuous feedback loop: it actively tracks redirects, deprioritises consistently erroring URLs, and updates its model of a site's structure across crawl cycles. Two decades of engineering have made Googlebot relatively efficient at staying current with how sites change.

AI training crawlers tend to work from static lists compiled at a point in time — often sourced from web archive datasets, index exports, or prior scraping runs — without the same live feedback mechanism. When a site restructures its URL scheme, consolidates pages, or removes content, Googlebot adjusts within a crawl cycle or two. An AI training crawler working from an older index might keep requesting those same dead URLs for months.

At 34.8%, GPTBot is fetching pages that don't exist on nearly a third of every crawl. That's not a minor inefficiency — it means a significant chunk of the AI crawler traffic in your logs is just the bot asking for things that aren't there anymore. Raw AI crawler visit counts should be treated with real scepticism until you've filtered by HTTP status code.

How fast is AI crawler traffic actually growing?

Despite the high 404 rates, AI crawler volume is expanding fast. GPTBot's share of verified bot traffic increased from 4.7% to 11.7% between July 2024 and July 2025 — a 2.5x jump in twelve months. ChatGPT-User, a separate bot that fires when a user is actively asking a question and the platform needs a live page, grew 2,825% year-over-year over the same window.

These two categories behave differently and matter differently for site owners. Training crawlers — GPTBot, ClaudeBot, CCBot, Amazonbot and others — account for roughly 80% of all AI bot activity. They collect pages for model training and might visit a page once then not return for weeks. Real-time retrieval bots, which trigger from an active user session, make up around 18% and are growing faster than training crawlers. They're also the only category with a plausible short-term path to sending you an actual visitor.

Most analytics setups don't separate these two categories — which makes it genuinely difficult to assess whether AI crawler activity represents meaningful reach or just infrastructure noise.

Is the crawling translating into real traffic?

Not in any proportion that would satisfy most marketing teams. Analysis of crawl-to-referral ratios in 2025 — how many page fetches a platform's crawlers made versus how many visits that platform sent back to the same sites — found extreme imbalances across major AI platforms. At peak, one platform was crawling at a rate of around 500,000 page fetches per referral visit returned. Even after these ratios moderated through the year, major AI crawlers were still running at tens of thousands of crawls per referral visit.

Your site can be thoroughly indexed by three separate AI training crawlers and generate near-zero referral traffic as a result. The content is being consumed — it's just not generating sessions. Treating training crawler traffic as a signal of AI discovery substantially overstates the business value of what those requests represent.

Does llms.txt change any of this?

llms.txt was proposed as a way for sites to give AI crawlers structured guidance — what to index, what to skip — in a format designed for language models rather than traditional search engines. Adoption has grown fast, but the effectiveness data is sobering.

As of June 2026, 8.7% of the world's top 1,000 websites publish an llms.txt file, up 8.8x from twelve months earlier. But 97% of those files receive zero AI crawler requests in a given 90-day window. Out of over 500 million AI bot visits tracked in one study period, just 408 requests fetched an llms.txt endpoint. The crawlers, in practice, aren't reading the file that most sites are publishing for them.

This doesn't mean you shouldn't deploy one — the implementation cost is trivial and the spec could gain real support as AI platforms iterate on how they discover and respect content signals. But right now it isn't a reliable lever for influencing how training crawlers treat your content.

What should site owners actually do about this?

Start with status code filtering before reading AI traffic headlines. If roughly a third of AI crawler hits are returning 404s, raw visit counts are significantly inflated. Filtering to 200-status requests gives a much more accurate picture of how much live content is actually being retrieved. A 100,000-request AI crawl session could easily be 65,000 real page loads and 35,000 dead ends.

Separate training crawlers from real-time retrieval bots in your analytics. They use distinct user-agent strings. Training crawlers (GPTBot, ClaudeBot, CCBot, Amazonbot) and retrieval bots — identifiable by suffixes like -User and -SearchBot across different AI platforms — should be tracked in separate buckets. Only retrieval bots have a direct line to a user session; conflating the two gives a misleading picture of AI reach.

Make your URL structure as stable as possible. The high 404 rate on AI training crawlers reflects stale URL lists that don't update when your site changes. Proper 301 redirects from moved pages, a current sitemap, and a stable URL scheme reduce the probability of your AI crawler traffic being wasted on content that no longer exists at those addresses. It won't fix a fundamentally stale training corpus, but it does limit the damage when your site architecture evolves.