Our site went down and every crawler noticed.
A bad deploy, a database restart, a host having a bad hour — while your origin returns 503, every crawler that visits writes that error down. And once you're back, some of them keep the old page.
What's actually happening
Crawlers don't wait for you to recover. They arrive on their own schedule, and if your server answers with a 5xx, that visit is spent. The page wasn't read, nothing new was indexed, and the AI crawler that came by to learn about your product left with an error.
One failed visit is rarely a disaster. Repeated ones are. Search engines slow down how often they crawl a site that keeps returning server errors, and pages that keep failing can drop out of the index altogether. AI assistants can only cite what their crawlers managed to read.
The opposite problem follows the fix. You ship new pricing or a rewritten page, and crawlers keep getting the copy that was cached before the deploy, until it expires or someone remembers to purge it.
5xx
What every crawler records while your origin is failing
Spent
A crawl that hits an error is not re-run on your schedule
Stale
Cached copies after a deploy, until something clears them
How Wrenda fixes it
Keep a good copy of every page crawlers read, hand it to bots when your origin fails, and swap it for a fresh one when you ship. People always see your real site.
- 01Every optimized or pre-rendered copy that was safe to cache is also kept for 30 days in a separate last-known-good store. Anything too personal to cache is never kept.
- 02An AI crawler, or a bot you pass through, reaches your origin and gets a 5xx. Instead of passing the error on, Wrenda serves the newest copy it holds — live cache first, then the 30-day store — marked
X-Wrenda-Status: origin-down-stale. - 03Human visitors get your origin's real response, error included. A 404 is never overridden, and if no copy exists the failure stands. Nothing is invented.
- 04Integration Health checks your origin on an hourly sweep. When a domain drops from healthy, your admins get an email, and another when it recovers.
- 05After a deploy, the hourly sitemap check sees your sitemap change, or your pipeline calls the purge hook, and the cached copies are cleared, so crawlers get the new pages instead of yesterday's.
What changes after you install
Crawlers get a page, not a 5xx
When a stored copy exists, an AI crawler meeting a failing origin gets a marked 200 instead of a wasted visit.
Your visitors see the truth
People always get your origin’s real response, so an outage is never hidden from the ones who need to know.
You hear about it
Seven checks on an hourly sweep, with an email to your admins when status regresses and when it recovers.
Fresh copies after you ship
A changed sitemap or a call from your deploy pipeline purges the cached copies, with no one remembering to do it.
Stay crawlable
Make your next outage a non-event for crawlers.
On by default for every domain. No code changes. Your visitors see nothing different.
Start 14-day trial14-day free trial — No credit card required