Why Are 97% of llms.txt Files Getting Zero Bot Requests?

Here's a number worth sitting with: 97% of all llms.txt files on the web got exactly zero bot requests in May 2026. Not one visit from GPTBot. Not one from ClaudeBot. Nothing. And yet adoption among the top 1,000 websites has grown from around 1% a year ago to 8.7% today — roughly a 5x jump in twelve months. If you've been tracking the AI SEO conversation, you've probably seen that adoption figure framed as momentum building. What tends to get less coverage is the other number sitting right next to it.

Where does that data come from?

Three independent studies converged in mid-2026. Rankability crawled the top 10,000 global websites in June 2026 and found 5.61% had a valid llms.txt, up from 1.04% in July 2025 — about 5.4x growth in twelve months. Separately, Ahrefs pulled a dataset of 137,000 domains and audited whether anything was actually requesting those files: 97% had received exactly zero requests from any bot or user during the measurement window. Acquia ran a fleet-wide audit across their entire hosting infrastructure — 400 million total requests — and found roughly 5,000 landing on /llms.txt. That's 0.001% of total traffic. The details of Shopify's platform-level rollout were first documented by Shopifreaks and later confirmed by the Weaverse development team in May 2026.

So why does the adoption headline look so strong?

The 8.7% adoption rate for the top 1,000 websites is a real number, but it's carrying a very particular asterisk. In late April and early May 2026, Shopify pushed llms.txt silently to every storefront on the platform — all 7 million of them — with no public announcement and no notification to merchants. By the time Shopifreaks reported on the rollout in early May, 78% of Shopify stores already had a file in place.

What's actually in the default Shopify-generated file? Your store name, a link to /collections/all, a contact page link, and endpoints for their new agentic commerce infrastructure. Merchants didn't write it, most don't know it's there, and very few have customized it beyond the defaults. That's platform adoption, not organic adoption — and the gap between the two is worth keeping in mind whenever you see the headline figures.

Strip out the platform-generated files and look at what happened through actual human choices, and the picture shifts. WordPress shows 8.7% adoption among sites in the top 1,000, which does reflect genuine decisions — someone installed a plugin or typed the file by hand. Among sites not running on a major CMS, the figures are considerably lower. Without the Shopify push, the June 2026 headline would likely have been something closer to 2–3% across the top 10k — real growth from a small base, but not quite the movement the coverage implied.

Why are AI bots not fetching the file?

Whether or not you have an llms.txt, the AI crawlers that drive the referral traffic you actually care about are almost certainly not fetching it. GPTBot, ClaudeBot, and PerplexityBot crawl HTML pages directly. In Acquia's hosting fleet analysis, 5,000 requests out of 400 million went to /llms.txt — and the vast majority of those came from SEO auditing tools and developer checks, not from production crawlers doing indexing runs.

So does having a well-structured llms.txt actually improve how often you show up in AI search results? As of mid-2026, there's no demonstrated causal link. Google confirmed in June 2026 that the file has no effect on Search rankings or AI Overviews — they simply don't use it. Research tracking sites with and without llms.txt across several months of AI search results found no statistically significant difference in citation rates. No major AI search provider has publicly stated they use llms.txt as a ranking or retrieval signal in production. The file isn't being read, and it isn't moving the needle.

Is there still a genuine case for shipping one?

Yes, actually — but probably not the one that gets talked about most. The original use case for llms.txt is narrower than the SEO story that's grown up around it: helping LLM-based coding assistants quickly navigate documentation for a library or API. That use case is real. Tools that help developers write code actively fetch these files when working with a codebase, and a well-structured file helps those tools understand what's where without crawling dozens of pages. If you're building a developer tool, an open-source library, or anything an AI coding assistant might want to understand, there's genuine value in a clean llms.txt.

There's also a reasonable forward-looking hedge. If a major AI search provider announces they're ingesting llms.txt as a discovery signal, the calculus changes overnight — and sites with a well-organized file already in place will be ahead. Given how little effort a good llms.txt actually takes to write, that option value is worth something.

What's harder to justify is treating it as a meaningful SEO lever right now. The bots don't read it. The answer engines don't weight it. And time spent on a file that production crawlers skip is time you're not spending on the things that demonstrably affect AI discoverability: HTML that's parseable without JavaScript execution, semantic structured markup, and building a presence in the sources that answer engines actually pull from.

What should you actually do?

If you're on Shopify, you already have an llms.txt — it was added without telling you. Worth a look. The default file points to /collections/all and a contact page, which probably isn't your most useful AI discovery path. A quick edit to surface your best product categories and support content makes the file at least useful if a bot does eventually check it.

For everyone else: add a simple llms.txt to the backlog if you're running a content-heavy or developer-facing site. Keep the scope narrow — a handful of links to your most semantically rich pages and a short description of what the site covers. Don't build an exhaustive content map; it won't be read.

And if someone on your team is pitching llms.txt as the missing piece for AI search visibility, the Acquia fleet numbers make for a useful reality check: 5,000 requests out of 400 million is not a signal worth building strategy around.