Why Is Your Website Invisible to AI Crawlers? Most Teams Are Getting This Completely Wrong

Here's a number worth sitting with: 97% of llms.txt files across 137,000 domains received zero AI crawler requests in May 2026 — not low traffic, actual zero. No bots, no humans, nothing crawling that file at all. And yet 8.7% of the world's top 1,000 websites now publish one, up 8.8x from twelve months ago.

So what's actually going on? Thousands of engineering hours are going into a signal that nobody is reading yet, while the more fundamental problem — AI crawlers can't see most of what you've built — gets almost no attention.

Wait, isn't llms.txt supposed to help AI systems discover your content?

That was the idea. Jeremy Howard proposed the format in late 2024: a Markdown file at your domain root listing your most important pages with brief context, designed to be processed directly by large language models. Documentation platforms picked it up fast — Vercel Docs, GitBook, Supabase, and Yoast all ship it as a default now. The adoption numbers climbed.

What didn't happen: any major AI search platform committing to actually read it as part of retrieval. The Ahrefs analysis of 137,000 domains found that where requests did arrive, 21.7% came from SEO audit tools, 14.9% from unidentified bots, and 13.1% from general web crawlers — not from the AI assistants the file was designed for.

Adoption by web tier tells the same story. Top 1,000 sites: 8.7% have llms.txt. Top 10,000: 5.6%. Broader 300,000-domain sample: 10.1%. The variation suggests adoption is driven by a cluster of early-adopter tech companies rather than any consistent uptake across the web. And even within that cluster, 97 out of every 100 files are sitting there uncrawled.

That doesn't mean you should rip the file out — but it does mean the effort going into it is outrunning any measurable benefit.

So what's the actual problem?

Rendering. Specifically, the fact that virtually no AI crawler executes JavaScript.

Vercel and MERJ analysed more than 500 million GPTBot fetches and found zero evidence of JavaScript execution. Even in the cases where GPTBot did download JavaScript files — roughly 11.5% of its requests — it treated them as text, not as instructions to build a page. Research across 23 major AI agents found that 69% of them cannot execute JavaScript at all. GPTBot, ClaudeBot, PerplexityBot, Meta's external web agent, ByteDance's Bytespider — none of these run client-side code.

What does that mean in practice? When one of these bots hits your page, it receives the raw HTML your server returned before any JavaScript ran. If your product descriptions, blog content, pricing tables, FAQs, or navigation items get built by JavaScript after that initial response — and for most React, Vue, or Angular apps, they do — the AI crawler doesn't see them. Those pages might as well not exist.

GPTBot makes approximately 4,200 requests per day to an average site. ClaudeBot runs at around 1,800. PerplexityBot at 980. These crawlers are hitting your pages constantly. If every one of those requests returns an HTML shell wrapping a JavaScript bundle and an empty div, they index empty pages.

Isn't this the same problem Google has?

Only partially. Google has built web rendering infrastructure that executes JavaScript as part of its indexing pipeline — and that infrastructure is what powers its AI search system too. So Google's AI assistant can see content that lives behind JavaScript. Every other major AI search product cannot.

This matters because the gap isn't evenly distributed. If you're relying on Google Search Console to tell you your pages are being indexed fine, that data doesn't say anything about whether PerplexityBot or any other AI crawler is seeing real content. They have separate pipelines. They don't render. And they're crawling your site thousands of times a day right now.

What does an AI crawler actually receive?

For a typical client-side rendered app, something like this: a root div element followed by a script tag pointing to your main JavaScript bundle. That empty div is what GPTBot indexes. Your entire product catalogue, every page description, every structured content block — locked inside that bundle, invisible.

And blocking makes this even harder to measure

There's a third variable worth flagging: 44.9% of major websites block at least one AI crawler in their robots.txt as of June 2026, with news publishers reaching 79%. A site can be blocking GPTBot entirely while simultaneously shipping an llms.txt file. Both decisions made without clear visibility into what the crawlers would actually have seen on the unblocked pages.

The sites most likely to care about AI crawler strategy — SaaS tools, content platforms, e-commerce built on React — tend to be the most JavaScript-heavy. The overlap between "team is thinking about AI discoverability" and "team has a rendering problem they don't know about" is probably very large.

What should you actually do here?

Start with your server logs, not your robots.txt. Filter for the AI crawler user-agent strings (GPTBot, ClaudeBot, PerplexityBot are the main ones). Check how many requests per day and which paths they're hitting. Then fetch those same URLs yourself with a bot user-agent and read what comes back. If the HTML body is mostly empty, you have a rendering problem — and no amount of llms.txt, schema markup, or meta tags will fix it.

If your site uses client-side rendering for its core content, the fix is server-side rendering or static generation. Next.js server-side rendering, Nuxt's SSR mode, or a dedicated pre-rendering layer for bot requests all work. This also helps traditional search crawlers that still have rendering queues.

For llms.txt: add it if your site has substantial documentation or reference content. The cost is low and it positions you for future support from AI platforms. Just don't mistake it for something that's working today.

The render gap is where the actual leverage is. If 69% of AI crawlers can't execute JavaScript and your site depends on JavaScript to deliver its content, that's the gap that's affecting every crawl, every day.