Can AI Crawlers Actually Read Your Website — or Are You Just Assuming They Can?

Here's a number that should give any site owner pause: 97% of websites that went to the trouble of publishing an llms.txt file — the format specifically designed to help AI crawlers navigate your content — received zero requests against that file in a one-month measurement window. Not a few. Almost all of them.

So the file existed. The crawlers visited. They just didn't read it.

That's the kind of finding that suggests a lot of effort is going into AI visibility work that isn't quite targeting the actual problem. Here's what the 2026 data says about how AI crawlers actually access web content — and where the real gaps are.

Where does the data come from?

The llms.txt figures are from Presenc AI's State of llms.txt 2026 research and Rankability's June 2026 crawl of the top 1,000 websites. JavaScript rendering statistics come from HybridRanking's July 2026 analysis and Onely's February 2026 study of JS content indexability. Adoption rates use Rankability's domain sample and a Tranco Top 10,000 analysis from May 2026.

How many sites have published llms.txt — and does it actually help?

The adoption curve looks decent on the surface. As of June 2026, 8.7% of the top 1,000 websites have an llms.txt file. Among the top 10,000, it's 5.86%.

Those numbers have grown fast — adoption was nearly zero in 2024. But growth in adoption and growth in effectiveness aren't the same thing.

Statistical analysis published in 2026 found no correlation between having an llms.txt file and how often a domain is cited by AI assistants. Not a weak correlation. No correlation. And the crawl-level data explains why: 97% of valid, published llms.txt files got zero crawler requests in a one-month sample window. The files are there. The crawlers aren't reading them.

This doesn't mean llms.txt is useless permanently — if more crawlers adopt the standard and begin actively requesting these files, the picture could shift. But right now, for most sites, the marginal benefit is close to zero.

The bigger problem: most AI crawlers can't render JavaScript

If you're building confidence in your AI visibility by publishing a machine-readable content index, you might also want to check what the crawlers actually see when they arrive at your pages.

69% of AI crawlers in 2026 cannot execute JavaScript at all. Googlebot is the only major crawler with full JavaScript rendering support. GPTBot downloads JavaScript files on roughly 11.5% of its page visits — ClaudeBot does so on roughly 23.8% — but neither actually executes them. They download the file and move on.

What this means in practice: any single-page application built with React, Next.js, Vue, Angular, or similar frameworks that relies on client-side rendering is serving AI crawlers an effectively blank page. The response body contains the skeleton HTML and a root div. That's it.

Onely's February 2026 analysis put a number on this: 42% of JavaScript-rendered content never gets indexed by AI systems. Not indexed slowly. Never indexed.

What does the failure actually look like in your logs?

The blank page problem is the most common failure mode, but not the only one. Three patterns show up regularly in crawler access logs:

Blank body on initial load. The HTML document contains nothing meaningful — product names, prices, descriptions, article text — until JavaScript executes. For AI crawlers that never run JS, this is an empty page with a title tag and not much else.

Lazy-loaded content below the fold. Even sites that have useful content in their initial HTML often defer secondary content, related products, or feature sections to lazy-load events that require scroll interaction. Crawlers that fetch once and leave don't trigger these. The content simply never arrives in the response they captured.

Authentication-gated rendering. Some apps render very little until a session check completes — even for public pages. The crawlers arrive before the check resolves, see a loading placeholder or near-empty shell, and move on.

If any of these describe your site, your AI visibility work is fighting the rendering stack before it even starts. llms.txt isn't going to help with that.

Are these problems fixable?

Yes — and the approaches are well-established at this point.

Server-side rendering or static generation. The cleanest fix for most modern frameworks is rendering pages on the server or at build time, so the crawler receives meaningful HTML on the first HTTP response. Next.js, Nuxt, SvelteKit, and similar frameworks all have SSR or static generation support. The question for most teams isn't whether it's possible — it's whether they've enabled it for the routes that matter.

Pre-rendering on bot detection. For sites where SSR isn't practical, a reverse proxy that detects bot user-agent strings and serves a pre-rendered HTML snapshot is a workable middle path. The crawler gets static content; real users get the interactive experience. It requires keeping snapshots fresh, but it handles the blank-page problem without a full architecture change.

Embed core content in the initial HTML. Even within a client-rendered app, it's usually possible to ensure that key information — the product name, the main body text, the essential facts — is in the initial HTML document rather than fetched dynamically. This isn't full SSR. It's discipline about what's in the shell versus what can load lazily.

What should you actually check first?

Pull up your own access logs and look at what your server returns when a page is fetched without JavaScript. A quick curl -A "GPTBot/1.1" https://yoursite.com/important-page and a scan of the response body will tell you immediately whether the crawlers are seeing your content or an empty shell.

The llms.txt investment is low-effort and may have real value as crawler support matures — keep it if you have it, but don't expect it to solve a rendering problem. AI crawlers can only work with what arrives in that first HTTP response. If that's a blank div, the rest of your AI visibility work is building on sand.