Is the robots.txt you wrote in 2023 still working for you in 2026?

One in four of the top 1,000 websites has explicitly blocked GPTBot from crawling them. Most of those sites did it to protect their content from being used in AI model training — and that's a completely legitimate reason. The problem is that GPTBot is not the bot that drives AI search citations. That's a different crawler, with a different user-agent string, operating on a different schedule for a different purpose. And the robots.txt rules written during the mass-blocking event of 2023 typically don't distinguish between the two.

So where does this data actually come from?

The blocking rate figures come from a large-scale analysis of robots.txt directives conducted across a major content delivery network's global infrastructure in Q1 2026, covering millions of domains. The traffic impact figures are from a working paper by Hangcheng Zhao (Rutgers Business School) and Ron Berman (The Wharton School), published April 2026, which combined daily traffic data, human browsing records from a panel survey, and robots.txt policy histories across a sample of publishers from October 2022 to June 2025. The llms.txt fetch rates come from log analysis tracking more than 500 million AI bot visits across a 90-day window in early 2026.

Who's actually blocking what right now?

When you scan DISALLOW rules across the web, GPTBot leads the list by a small margin:

Those percentages look modest at the domain level, but they translate to hundreds of thousands of sites. At the top end of web traffic, the picture is starker: 25% of the top 1,000 websites now block GPTBot, up from roughly 5% when the crawler was first introduced in mid-2023. Among news publishers specifically, the figure reaches 79% — eight in ten of the world's biggest news websites have added blocking rules for at least one AI training crawler.

Here's the thing most of those rules get wrong

Every major AI provider that operates web crawlers runs at least two distinct bots: one for collecting content that goes into model training, and a separate one that powers real-time search indexing and citations. These bots use different user-agent strings. They're controlled independently in robots.txt. Blocking one does not block the other.

GPTBot collects content that may eventually be used to train AI foundation models. OAI-SearchBot is the indexer that determines whether a page surfaces in AI search answers. When a site adds Disallow: /GPTBot to their robots.txt, they're opting out of training data collection — but OAI-SearchBot keeps crawling, and those pages can still appear in AI-generated responses.

The same principle applies across the board. ClaudeBot crawls for training. Its search-indexing counterpart uses a different user-agent string and won't be affected by a ClaudeBot block. PerplexityBot, which powers citations in Perplexity search, is a retrieval crawler — blocking training bots from the same provider won't prevent PerplexityBot from indexing your pages.

Most of the mass blocking that happened in 2023 and 2024 didn't make this distinction, because the distinction wasn't widely understood at the time. Sites saw GPTBot in their logs, read about model training concerns, and added a DISALLOW rule for the most recognisable user-agent name they could find. Those rules are largely still in place, unchanged.

And what actually happened when publishers blocked?

The Rutgers and Wharton research tracked what happened to publisher traffic after sites added robots.txt rules targeting AI training crawlers, analysing more than two years of daily traffic data alongside human browsing panel records. The result wasn't what most blocking sites expected:

Publishers that blocked AI crawlers via robots.txt saw roughly a 7% decline in weekly visits within the six weeks following the blocking decision. Human browsing dropped by a similar amount. That's a meaningful hit for sites that were blocking specifically to protect their content and reader relationships.

The mechanism is indirect but consistent with how AI-assisted discovery works. When AI systems recommend or cite sources, they drive downstream traffic through search results and direct links. Blocking training crawlers can disrupt that pathway — especially if search and retrieval crawlers share infrastructure with training crawlers and partially respect broad blocking rules. The practical result is that many sites took on both the content risk they were trying to avoid and a traffic cost they weren't expecting.

Does llms.txt solve any of this?

llms.txt was designed as a structured way to tell AI systems what your site contains and how to interpret it — a purpose-built signal for the AI era, distinct from robots.txt's blunter allow/block mechanism. Adoption has been real: 8.7% of the Tranco top-1,000 sites now publish an llms.txt file, and the count across the web has grown significantly over the past year.

But is anyone actually reading it? Log analysis covering 500 million AI bot visits over 90 days in early 2026 found that just 408 requests targeted a llms.txt file. That's less than one in a million AI crawler requests. GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and Google-Extended all crawl past it and parse HTML directly, which is the same thing they've always done.

The file does have genuine value for one specific use case: developer-facing sites where AI coding tools need to understand your APIs and documentation. When someone uses an AI coding tool and asks it to build an integration with your product, a well-structured llms.txt can be the difference between the tool generating working code and hallucinating non-existent endpoints. That's a real, measurable benefit — and it's why API-first companies have adopted the format quickly.

For general AI search visibility? The file isn't the lever you'd want it to be. As of mid-2026, no major AI search provider has publicly committed to reading or acting on llms.txt as a ranking or indexing signal in their production systems. Publishing it costs nothing and it's worth doing for the developer-tooling use case. Just don't count on it to influence how GPTBot or OAI-SearchBot treat your pages.

So what does a sensible policy actually look like now?

The clearest reframe is to stop treating all AI crawlers as the same thing and start grouping them by what they're actually doing.

Search and retrieval bots — OAI-SearchBot, PerplexityBot, the indexing agents from the same providers as training crawlers — are the ones that drive citations and referral traffic. These are worth explicitly allowing in robots.txt. They have much better crawl-to-referral ratios than training bots, and blocking them is a direct trade-off with AI search presence. If you blocked a major provider's training bot in 2023 using a catchall rule, there's a reasonable chance the search bot got caught in the same net.

Training crawlers — GPTBot, ClaudeBot, CCBot and similar — are a separate decision. You can disallow them if you have policy concerns about training data use. But understand the trade-off: you lose the indirect benefits of being represented in those models (brand familiarity, your terminology appearing in AI responses, your domain being cited where the model draws on prior training). That trade-off may be worth it, or it may not. The Rutgers/Wharton data suggests it often comes with a traffic cost that most blocking sites didn't anticipate.

Everything else — agentic crawlers, newly announced bots, the long tail of scrapers claiming AI purposes — can typically be rate-limited and monitored without blanket blocking.

The robots.txt you haven't touched since 2023 is probably not making the training/search distinction. More than 40% of B2B sites still run blocking rules that predate this split becoming mainstream. A quick audit against the current list of known AI user-agents — checking which bots you're allowing, which you're blocking, and whether your search indexers are explicitly permitted — is worth doing before the next time you wonder why your content isn't appearing in AI search results.

The question has never really been "should AI crawlers be allowed on my site?" Most sites would say yes to that in principle. The real question is which crawlers, for what purpose, under what rules — and most robots.txt files from 2023 are answering something simpler and blunter than that.