Your Site Does SSR. So Why Are 46% of Your Pages Still Invisible to AI Crawlers?

If you asked most ecommerce engineers whether their Next.js or Shopify store is "AI-ready," they'd probably say yes. They're using SSR. They've got structured data in the page. Some have even published an llms.txt. So why does Adobe's latest analysis of over a trillion retail site visits show that up to 46% of retailer pages aren't readable by AI crawlers at all?

The uncomfortable answer is that server-side rendering doesn't fix everything — and the failure modes that survive SSR are the ones that most engineering teams haven't thought about yet.

Where the data comes from

The 46% figure comes from Adobe Digital Insights' Q1 and Q2 2026 retail AI traffic reports, which analyzed over one trillion visits to US retail websites. AI traffic conversion data comes from the same study. The JavaScript rendering behaviour analysis — including a dataset of over 500 million GPTBot requests — comes from separate third-party log and network studies published in 2026.

Why does this even matter right now?

Because the stakes have changed fast. In Q1 2026, AI-driven traffic to US retail sites grew 393% year over year. During the 2025 holiday season, that figure hit 693%. And the traffic that does get through converts well — AI referrals converted 42% better than non-AI traffic in March 2026, and shoppers arriving from AI sources spent 53% more time on site.

The gap between AI readability and AI traffic volume is widening. If up to 46% of your pages are invisible and AI traffic is growing at 393% annually, the opportunity cost of that readability gap is compounding at the same rate.

So what exactly does "not readable by AI" mean?

When Adobe says a page is "not readable by machines," it means an AI crawler requesting that URL receives HTML from which meaningful content cannot be extracted. That happens in four main ways.

JavaScript-dependent rendering. The main culprit. 69% of AI crawlers — including GPTBot, PerplexityBot, and the majority of other active AI bots — cannot execute JavaScript at all. An analysis of over 500 million GPTBot requests found zero evidence of JavaScript execution. GPTBot downloads JS files in roughly 11.5% of crawl sessions — but downloading and running are two different things, and it never actually runs the code.

If your product title, price, availability, or description exist only in a JS-rendered component rather than in the HTML the server sends, an AI crawler sees nothing there.

next/dynamic with { ssr: false }. The most common accidental trap in Next.js projects. When you write next/dynamic(() => import('./ProductDetails'), { ssr: false }), you're explicitly telling Next.js not to render that component on the server. That's fine for interactive widgets that don't need to be indexed, but it's a problem when the component contains product content. A product details panel loaded this way simply doesn't exist in the HTML response a crawler receives. Developers often do this to avoid hydration mismatches or to defer heavy components, without realising they've removed the content from the initial render entirely.

Lazy loading with Intersection Observer. Content loaded via the Intersection Observer API — expanding specification panels, "load more" review sections, or below-the-fold content that triggers on scroll — only loads when a user scrolls to it in a browser. An AI crawler requests the URL and receives the initial HTML; it doesn't scroll and it doesn't wait. That content never appears.

Client-fetched dynamic data. Reviews, personalised recommendations, real-time stock levels, and "frequently bought together" sections are commonly populated via API calls in useEffect hooks or client-side data-fetching libraries. Even on a fully SSR site, these components often send their data request from the browser after hydration. What the server sends is a loading skeleton or an empty container. That's what the crawler indexes.

How do you find out which of your pages have this problem?

The fastest diagnostic is a curl request with an AI bot user-agent. Run this against any page you're concerned about:

curl -A "GPTBot/1.0" https://yoursite.com/product/your-slug

Look at what's in the response body:

• Is the product title actually in the HTML, or just in a JS bundle?
• Is the price rendered server-side, or populated client-side?
• Are descriptions and specifications there, or behind a lazy-load trigger?
• Is there an empty mount point like <div id="root"></div> instead of content?

Compare that curl output to what you see in a browser. The gap between those two views is your AI visibility gap.

For Next.js specifically, the clearest signal is your React Server Component boundary. Any component marked 'use client' that fetches its own data via useEffect or a data-fetching hook is sending an empty or skeleton state in the server response. Any component running as a Server Component has a much better chance of appearing in the initial HTML the crawler receives.

What should you actually do about it?

The fix usually isn't a full rewrite. In most cases you're looking at moving data fetching from client-side hooks into server-side functions, removing { ssr: false } from content-carrying components, and making sure product title, price, description, and key specifications are present in the initial HTML response — even if the rest of the page hydrates client-side.

Start with your highest-value pages and a simple curl test. If the content isn't in the curl response, that page is invisible to most AI crawlers regardless of how it looks in a browser.

The broader point is that "AI readability" isn't a one-time fix. Every lazy-load optimisation, client-side data fetch, or { ssr: false } dynamic import added to a product page is a potential new blind spot. The same discipline that applies to server-side rendering for traditional search applies here — with the added challenge that the list of crawlers you need to serve has grown from one to more than a dozen active AI bots, all representing a traffic segment growing at several hundred percent annually and converting better than almost any other referral source.