Crawlable When Your Origin Isn't
When your origin returns a server error, the crawlers that reach it get the last good optimized or pre-rendered copy Wrenda holds instead of a 5xx. Your visitors still see the real error, and Integration Health tells you something is wrong.
Built in
5xx FallbackLast-Known-Good StoreBots OnlyIntegration HealthSitemap WatchA bad hour shouldn't cost a crawl
Wrenda already holds a good copy of the pages crawlers ask for. When your origin fails, it uses it.
Feature 01
Stale-Serve -- The Last Good Copy for Bots
When an AI crawler or a bot you pass through reaches your origin and it answers with a 5xx, Wrenda serves the optimized or pre-rendered copy it already holds, as a normal 200. Nothing is invented: if there is no stored copy, the crawler gets your origin's error as it is.
- Triggered by any 5xx from your origin
- A 404 or other 4xx is never overridden
- Bots only: people always get the real error
- Live cache first, then the 30-day store
- On by default, one switch per domain
Per-domain setting
{
stale_on_origin_error: true, // default
trigger: '5xx', // 500–599 only
served_to: 'bots' // never humans
}What happens on a 5xx
Origin answers 5xx
A 4xx is never overridden
Is it a bot?
A person gets the real error
Check the live cache
Freshest copy first
Then last-known-good
Up to 30 days old, with Age
Response headers
X-Served-By: wrenda-stale X-Wrenda-Status: origin-down-stale X-Wrenda-Stale-Source: last-known-good Age: 93412 Cache-Control: public, max-age=300
Feature 02
Last-Known-Good -- 30 Days of Cover
The live cache expires in an hour for optimized pages and a day for pre-rendered ones. A real outage can outlast both. So every copy that was safe to cache is also kept for 30 days, and read only when your origin can't answer.
- Written on every fresh copy -- Each new optimize or pre-render replaces the stored one
- Privacy inherited -- Only kept when the normal cache write was allowed
- Survives setting changes -- Not tied to your current optimization set
- Never extended -- Serving a copy does not reset its 30 days
Two copies, two jobs
Live cache
What crawlers get every day. One hour for optimized pages, 24 hours for pre-rendered, configurable per content rule. Purged when your sitemap changes or your purge hook fires, and invalidated when you change an optimization.
Last-known-good store
Read only when your origin answers with a 5xx and the request is from a bot. Kept 30 days. A purge or a settings change leaves it in place, because during an outage any good copy beats an error.
Your origin
Still the source of truth. Human visitors always reach it directly, errors included, and every fresh copy Wrenda keeps starts as a good answer from it.
Know before your rankings do
Serving bots a good copy buys you time. Integration Health makes sure you spend it fixing the problem.
Feature 03
Seven Checks, One Status
Integration Health answers one question for every domain: is Wrenda still correctly in front of it? Seven checks follow the path a request takes, from DNS to your origin to the crawlers arriving, and roll up into one status you can read at a glance.
- Down, degraded or healthy -- Any failure in DNS, certificate, route or origin means down
- Hourly sweep -- Verified domains are re-checked in the background, oldest first
- Regression email -- Admins hear when status drops, and again when it recovers
- Fix button -- Re-creates a missing route or re-runs verification where Wrenda can
Alerts are de-duplicated to once a day per domain, so a flapping origin sends one regression and one recovery email, not a flood. Run a live check from the dashboard any time.
Read the health docsThe seven checks
| Check | What it verifies | If it fails |
|---|---|---|
| DNS CNAME | Domain resolves to Wrenda | Down |
| Certificate validation | DCV delegation CNAME present | Degraded |
| Ownership TXT | Verification record present | Degraded |
| SSL certificate | Certificate active, with expiry | Down |
| Worker route | Traffic reaches the proxy | Down |
| Origin | Answers, with status and latency | Down |
| Crawler traffic | Recent bots, 24h cache-hit ratio | Degraded |
For CDN and web-server integrations the DNS, certificate and route checks reflect only what Wrenda can see, and the route check is skipped.
From error to answer
The same four steps for every crawler request that meets a failing origin.
Step 01
Your origin answers 5xx
An AI crawler or a passed-through bot reaches your origin and gets a server error back. A 4xx is a correct answer and passes straight through.
Step 02
Bots only
If the request is from a person, the error is returned exactly as your origin sent it. If the switch is off for the domain, the same happens for bots.
Step 03
Find the newest good copy
Wrenda checks the live cache, then the 30-day last-known-good store, for a pre-rendered or optimized copy of that URL.
Step 04
Serve it, marked, or let the error stand
A copy is served as a 200 with X-Wrenda-Status: origin-down-stale and a five-minute Cache-Control. No copy means the crawler gets your origin’s error.
The other half: fresh copies
Outage cover keeps old copies around. When you ship, crawlers should get the new pages instead.
Sitemap Watch
Every hour Wrenda checks the sitemap of each verified, active domain. When it is newer than last time, the domain’s cached copies are purged so crawlers get fresh ones.
- Uses your <lastmod> dates
- Follows up to 10 child sitemaps
- Nothing to install
Purge Hook
A private URL your deploy pipeline calls when a build succeeds. Purge the whole domain, or just the pages a CMS publish changed.
- Vercel, Netlify, GitHub Actions
- Whole domain or listed URLs
- Token-protected, rate-limited
Cache Warming
Re-populate the pages you care about most from your sitemap, so the first crawler after a purge already gets a cached copy.
- Manual or scheduled runs
- Per-run history
- Driven by your sitemap
Same URL, two honest answers
A crawler gets the stored copy, clearly marked. A person gets your origin's real response.
# An AI crawler while the origin is returning 503 $ curl -I -H "User-Agent: GPTBot/1.0" https://yoursite.com/pricing HTTP/2 200 content-type: text/markdown; charset=utf-8 x-served-by: wrenda-stale x-wrenda-status: origin-down-stale x-wrenda-stale-source: last-known-good x-ai-optimized: true cache-control: public, max-age=300 age: 93412 # A person, same URL, same moment $ curl -I -A "Mozilla/5.0 Chrome/120" https://yoursite.com/pricing HTTP/2 503 x-served-by: wrenda-fast
30 days
Last-known-good retention
7
Integration checks per domain
Hourly
Health sweep and sitemap watch
Outage Protection FAQs
What triggers the fallback, which crawlers it covers, and how you hear about an outage.
Any 5xx response (500 to 599) from your origin. A 404 or any other 4xx is your origin answering correctly, and Wrenda never overrides it, so a page you deleted stays deleted. The fallback does not cover an origin that never answers: if the connection fails or times out, the crawler gets an error rather than a stored copy.
Because they are the people who need to know. Hiding an outage from your visitors, or from you when you check your own site, would turn a short incident into a long one. The fallback is for crawlers only: a person always gets your origin’s real response.
Any bot whose request goes through to your origin while it is failing: AI crawlers whose optimized copy has expired, and bots you pass through. Crawlers on a pre-render rule are served the cached render as usual while it is fresh (24 hours by default) without your origin being asked at all. Once that render expires, your origin’s response reaches them.
Up to 30 days. Wrenda tries the live cache first, because it is the freshest copy, then the last-known-good store. A copy from the store carries an Age header so the crawler can judge it for itself. Serving a stored copy never extends its life, and every fresh optimize or pre-render replaces it.
No. A copy is kept for 30 days only when the normal cache write succeeded, and that write is gated by the same privacy check that decides whether a page is safe to cache at all. If a response was too personal to cache, it is too personal to keep.
No. The live cache is purged when your sitemap changes or your purge hook fires, and invalidated when you change an optimization, so crawlers get fresh copies. The last-known-good store is kept apart from both, because during an outage any good copy of the page beats an error. It is replaced the next time a fresh copy is made.
It is on by default for every domain, with one switch per domain on the domain’s settings page: “Serve the last good copy when your origin is down”. Admins and editors can change it. Stored copies served during an outage are not counted against your monthly allowance.
Integration Health checks your origin as one of its seven checks. When the hourly sweep sees a domain go from healthy to degraded or down, you get an in-app notification and your admins get an email, at most once a day per domain, plus a recovery email when it comes back.
Get Started
Don't hand crawlers a 5xx
Outage cover is on by default for every domain. Integration Health watches the rest.
Start 14-day trial14-day free trial — No credit card required