Is That Really GPTBot?
Anyone can claim to be a crawler. Wrenda checks each claim against the IP ranges the vendor publishes, shows you how many failed, and lets you decide what a fake gets.
Ranges checked
GoogleBingOpenAIPerplexityCheck the claim, then decide
Your rules decide what a crawler gets from its User-Agent. Verification makes sure the crawler is who it says it is.
Feature 01
The Check -- User-Agent vs Source IP
Google, Bing, OpenAI and Perplexity each publish the IP ranges their crawlers use. When a request claims to be one of them, Wrenda compares the address it actually came from against that vendor's list and records one of three verdicts.
- IPv4 and IPv6 ranges, straight from each vendor's own files
- Search crawlers and AI crawlers: Googlebot, Bingbot, GPTBot, PerplexityBot
- Ranges refreshed daily and held at the edge
- A vendor outage keeps the last good copy, never a false fail
- Only checks requests where a fake would cost you: optimize and pre-render
One check, illustrated
{
user_agent: 'GPTBot/1.1',
source_ip: '203.0.113.42',
claimed_vendor: 'openai',
checked_against: ['gptbot.json', 'searchbot.json', 'chatgpt-user.json'],
verdict: 'unverified'
}Three verdicts
| Verdict | Meaning | Result |
|---|---|---|
verified | IP is inside the vendor's published ranges | Served by your rule |
unverified | Vendor publishes ranges; IP is outside them | Your chosen mode applies |
no ranges | Vendor publishes no range file, e.g. ClaudeBot | Served by your rule |
Any other crawler, and every human visitor, is not checked and is served exactly as before.
Why “no ranges” is not a fail
Anthropic publishes no IP range file for ClaudeBot. Treating that as a spoof would turn away a real AI crawler over a file Anthropic hasn't published. Wrenda only ever acts on a claim it checked and that failed.
Feature 02
The Evidence -- See It Before You Enforce
Every checked request is logged with its verdict. Each domain's dashboard turns that into one number: what share of checked crawler claims failed. Nobody is asked to choose Block without seeing what it would have affected.
- Verdict counts -- Verified, unverified and no-ranges totals for the domain
- Failure share -- Unverified claims as a share of every claim that was checked
- The culprits -- The user agents sending the most unverified claims, with counts
- Recent window -- The last 7 days by default, up to 14
When a claim fails
Observe (default)
Record the verdict and serve exactly as before. Nothing changes for any crawler. Start here and build up the evidence.
Demote
The unverified claim still gets your site, the same page a visitor gets, but not the AI-optimized or pre-rendered copy you paid to produce.
Block
The unverified claim gets a 403. Strongest protection, least forgiving if a vendor changes its ranges, so the dashboard asks you to confirm first.
One check, before anything is served
The check runs at the edge, after your rules pick a treatment and before Wrenda serves it.
Step 01
A request claims to be a crawler
The User-Agent says Googlebot, Bingbot, GPTBot or PerplexityBot, and your rule for it is optimize or pre-render: the treatments a fake claim would take from you.
Step 02
The source IP is compared
The address the request came from is matched against the claimed vendor's published ranges, already stored at the edge. No vendor is contacted while the crawler waits.
Step 03
The verdict is recorded
Verified, unverified or no ranges is written to the request log. That log is what the dashboard report is built from.
Step 04
Your mode decides the response
Observe changes nothing. Demote serves your normal page instead of the optimized copy. Block returns a 403. Verified and uncheckable requests are always served by your rules.
What is checked, and what isn't
Verification is only as good as the vendor's published list, so it is honest about where those lists exist.
Checked against published ranges
Four vendors publish machine-readable IP range files. Claims to be any of their crawlers are verified.
- Google: Googlebot, GoogleOther, Storebot-Google, Google-InspectionTool
- Bing: Bingbot, AdIdxBot, BingPreview
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- Perplexity: PerplexityBot, Perplexity-User
Recognized, never penalized
Anthropic publishes no range file, so its crawlers are recorded as "no ranges" and served by your rules as normal.
- ClaudeBot
- Claude-User and Claude-Web
- anthropic-ai
- Never demoted, never blocked
Built to fail open
A check that cannot be completed is never counted as a failure. Real crawlers keep getting your content.
- Vendor file unreachable: last good copy is used
- No copy yet: "cannot check", not "failed"
- Any internal error: served as normal
Pretend to be GPTBot
Your laptop is not in OpenAI's ranges, so a GPTBot claim from it fails the check. With GPTBot set to optimize on an active domain, here is what it gets in each mode.
# Claim to be GPTBot from an ordinary connection $ curl -I -H "User-Agent: GPTBot/1.1" https://yoursite.com/ # Mode: Block HTTP/2 403 content-type: text/plain; charset=utf-8 x-wrenda-bot-verified: unverified # Mode: Demote (your normal page, not the optimized copy) HTTP/2 200 x-crawler-action: pass-through x-ai-optimized: false # Mode: Observe (served as before; the verdict goes in the report)
4
Vendors with ranges checked
3
Modes: observe, demote, block
Daily
Range files refreshed
Where verification fits
Crawler Verification FAQs
What is checked, what a fake crawler gets, and why real crawlers are never caught out.
Because anyone can write one. A User-Agent is a line of text the requester chooses, so a scraper can send "GPTBot" or "Googlebot" from any server. Wrenda decides what a crawler is served from that line, which is why it also checks where the request actually came from: the source IP is compared against the IP ranges the claimed vendor publishes.
Not by default. Every domain starts in Observe mode, which records the verdict and changes nothing. Enforcement only ever acts on a claim that was checked and failed. If a vendor's range file cannot be fetched, Wrenda keeps using the last copy it has, and if it has no copy at all the request is treated as "cannot check", never as a failure. The one real risk is a vendor adding new ranges before Wrenda's daily refresh picks them up, which is why Demote is the safer choice if you are unsure, and why choosing Block asks you to confirm first.
Anthropic does not publish a machine-readable IP range file for ClaudeBot, so there is nothing to check it against. Wrenda recognizes Anthropic's crawlers and records them as "no ranges", which is not a failed check. They are never demoted or blocked by this feature. The trade-off is plain: a scraper pretending to be ClaudeBot cannot be caught by this check either.
Observe: exactly what it got before, so you can measure first. Demote: your normal site, the same page a visitor would get, but not the AI-optimized or pre-rendered copy. Block: a 403 Forbidden response. In every mode, verified crawlers and anything that could not be checked are served by your rules as normal.
Requests that claim to be a crawler from Google, Bing, OpenAI or Perplexity, on a rule set to optimize or pre-render, on a domain in active mode. Those are the requests where a fake claim costs you something. Pass-through and blocked rules, domains in shadow or paused mode, other crawlers and human visitors are not checked.
The vendor range files are fetched by a daily background job and stored at the edge, so a check is a lookup against data already on hand. Wrenda never contacts Google, OpenAI or any other vendor while a crawler is waiting for your page.
Not automatically. Crawler analytics count requests by the user agent they arrived with. The verification report sits alongside them and shows what share of checked crawler claims failed and which user agents sent them, so you can read your crawler numbers with that in mind.
Every plan, including the 14-day trial. Anyone on your team can see the report; admins and editors can change the mode.
Get Started
Serve the real crawlers
Verification runs in Observe mode as soon as a domain goes active. Read the evidence, then choose how strict to be. Included on every plan.
Start 14-day trial14-day free trial — No credit card required