Install

Web server & CDN snippets

When DNS can't change, your server or CDN keeps serving the site and forwards only bot requests to Wrenda. The snippets below are generated by the same code the dashboard uses, with example.com as the domain — the wizard renders them with yours.

How it works

Every snippet does one thing: test the User-Agent against Wrenda's canonical bot pattern and, on a match, proxy the request to https://api.wrenda.ai/proxy/{your-domain}{path}. Wrenda then classifies the bot against your rules and applies optimize, pre-render, pass-through or block, exactly as it would in DNS proxy mode. Because the classification happens on Wrenda's side, the pasted pattern never needs updating when new crawlers appear — Wrenda's registry is updated centrally.

The snippets forward these headers so Wrenda can log and route correctly:

Host
api.wrenda.ai (and the same TLS SNI). Sending your own hostname here makes Cloudflare misroute the request — Wrenda reads your domain from the path.
Authorization
Bearer + an API key from your workspace — see below.
X-Forwarded-Host
The hostname the visitor requested.
User-Agent
The original crawler user agent, unchanged.

When Wrenda fetches your origin it adds X-Wrenda-Origin-Fetch: 1, and every snippet skips Wrenda for requests that carry it. Search bots are fetched with their own user agent, so without that check a snippet running on your origin would send Wrenda's fetch straight back to Wrenda in a loop. Keep the check if you adapt a snippet.

Server snippets

In the dashboard these are shown on the final wizard step for CDN Integration and Web Server Proxy, with your domain substituted. Replace example.com and YOUR_WRENDA_API_KEY if you copy from here.

Nginx

nginx.conf
# Wrenda AI Crawler Proxy Configuration for example.com
# Add this to your Nginx configuration.
#
# Routes ALL bot traffic to Wrenda. Wrenda classifies the bot and applies
# the right action (optimize / pre-render / pass-through / block) based
# on routing templates configured in your dashboard.
#
# Replace YOUR_WRENDA_API_KEY with an API key from
# Settings → API & MCP (a read-scoped key is enough).

# Detect bots that should be routed to Wrenda (http {} context)
map $http_user_agent $wrenda_bot {
    default 0;
    "~*GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot" 1;
}

# Wrenda fetching your origin sends X-Wrenda-Origin-Fetch; never route
# that back to Wrenda.
map $http_x_wrenda_origin_fetch $is_wrenda_crawler {
    ""      $wrenda_bot;
    default 0;
}

server {
    # ... your existing server configuration ...

    # Needed because proxy_pass below is resolved per request.
    resolver 1.1.1.1 1.0.0.1 valid=300s ipv6=off;
    resolver_timeout 5s;

    location / {
        # Forward bot traffic to Wrenda (handled in @wrenda below)
        error_page 418 = @wrenda;
        if ($is_wrenda_crawler) {
            return 418;
        }

        # Your existing configuration for non-bot traffic
        # proxy_pass http://your-origin;
    }

    location @wrenda {
        proxy_pass https://api.wrenda.ai/proxy/example.com$request_uri;
        proxy_set_header Host api.wrenda.ai;
        proxy_set_header Authorization "Bearer YOUR_WRENDA_API_KEY";
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header User-Agent $http_user_agent;
        proxy_ssl_server_name on;
        proxy_ssl_name api.wrenda.ai;
    }
}

Apache

httpd.conf / .htaccess
# Wrenda AI Crawler Proxy Configuration for example.com
# Add this to your Apache VirtualHost or .htaccess.
# Requires: mod_proxy, mod_proxy_http, mod_ssl, mod_rewrite, mod_headers
#
# Routes ALL bot traffic to Wrenda. Wrenda decides per-bot what to do
# based on routing templates configured in your dashboard.
#
# Replace YOUR_WRENDA_API_KEY with an API key from
# Settings → API & MCP (a read-scoped key is enough).

# Wrenda is reached over HTTPS (VirtualHost only — not allowed in .htaccess)
SSLProxyEngine on

RewriteEngine On

# Forward bot traffic to Wrenda — but not Wrenda's own fetches of your
# origin, which carry X-Wrenda-Origin-Fetch
RewriteCond %{HTTP:X-Wrenda-Origin-Fetch} ^$
RewriteCond %{HTTP_USER_AGENT} (GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot) [NC]
RewriteRule ^ https://api.wrenda.ai/proxy/example.com%{REQUEST_URI} [P,L,E=WRENDA:1]

# Proxy settings for Wrenda requests (mod_proxy adds X-Forwarded-Host itself)
ProxyPreserveHost Off
ProxyPassReverse / https://api.wrenda.ai/proxy/example.com/

# Authenticate to Wrenda — only on requests routed above
RequestHeader set Authorization "Bearer YOUR_WRENDA_API_KEY" env=WRENDA

# Enable these Apache modules if not already enabled:
# a2enmod proxy proxy_http ssl rewrite headers

Caddy

Caddyfile
# Wrenda AI Crawler Proxy Configuration for example.com
# Add this to your Caddyfile.
#
# Routes ALL bot traffic to Wrenda. Per-bot routing decisions happen
# server-side via routing templates in your dashboard.
#
# Replace YOUR_WRENDA_API_KEY with an API key from
# Settings → API & MCP (a read-scoped key is enough).

example.com {
    @wrenda_crawler {
        header_regexp User-Agent (?i)(GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot)
        # Wrenda fetching your origin — never route it back to Wrenda
        not header X-Wrenda-Origin-Fetch *
    }

    # Forward bot traffic to Wrenda
    handle @wrenda_crawler {
        rewrite * /proxy/example.com{uri}
        reverse_proxy https://api.wrenda.ai {
            header_up Host {upstream_hostport}
            header_up Authorization "Bearer YOUR_WRENDA_API_KEY"
            header_up X-Forwarded-Host {host}
        }
    }

    # Your existing configuration for non-bot traffic
    handle {
        # reverse_proxy your-origin:port
    }
}

Cloudflare Worker

worker.js
// Wrenda AI Crawler Proxy — Cloudflare Worker
// Deploy this worker and create a route for example.com/*
//
// This forwards ALL bot traffic to Wrenda. Wrenda then classifies the
// request and applies the right action (optimize / pre-render /
// pass-through / block) based on the routing templates you configure
// in your dashboard. You don't need to update this file when new AI
// crawlers emerge — Wrenda's bot list is updated server-side.
//
// Store your API key (Settings → API & MCP; read scope is enough) as a secret:
//   npx wrangler secret put WRENDA_API_KEY

const WRENDA_BOT_PATTERN = /GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot/i;
const WRENDA_PROXY_URL = 'https://api.wrenda.ai/proxy/example.com';

export default {
  async fetch(request, env) {
    const userAgent = request.headers.get('User-Agent') || '';

    // Forward all bot traffic to Wrenda — let Wrenda decide what to do.
    // Requests carrying X-Wrenda-Origin-Fetch are Wrenda fetching your
    // origin; sending those back to Wrenda would loop.
    if (WRENDA_BOT_PATTERN.test(userAgent) && !request.headers.has('X-Wrenda-Origin-Fetch')) {
      const url = new URL(request.url);
      return fetch(WRENDA_PROXY_URL + url.pathname + url.search, {
        method: request.method,
        redirect: 'manual',
        headers: {
          'Authorization': `Bearer ${env.WRENDA_API_KEY}`,
          'User-Agent': userAgent,
          'Accept': request.headers.get('Accept') || '*/*',
          'Accept-Language': request.headers.get('Accept-Language') || '',
          'X-Forwarded-Host': url.hostname
        }
      });
    }

    // Non-bot traffic: pass through to origin untouched.
    return fetch(request);
  }
};

Express.js

server.js
// Wrenda AI Crawler Proxy Middleware for Express.js
// npm install http-proxy-middleware@3
//
// Forwards ALL bot traffic to Wrenda. Per-bot routing decisions happen
// server-side via routing templates in your dashboard.
//
// Set WRENDA_API_KEY in your environment to an API key from
// Settings → API & MCP (a read-scoped key is enough).

const { createProxyMiddleware } = require('http-proxy-middleware');

const WRENDA_BOT_PATTERN = /GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot/i;

const wrendaProxy = createProxyMiddleware({
  target: 'https://api.wrenda.ai',
  changeOrigin: true, // sends Host: api.wrenda.ai
  pathRewrite: (path) => `/proxy/example.com${path}`,
  on: {
    proxyReq: (proxyReq, req) => {
      proxyReq.setHeader('Authorization', `Bearer ${process.env.WRENDA_API_KEY}`);
      proxyReq.setHeader('X-Forwarded-Host', req.headers.host || 'example.com');
    }
  }
});

// Apply middleware BEFORE your routes
app.use((req, res, next) => {
  const userAgent = req.headers['user-agent'] || '';
  // Requests carrying X-Wrenda-Origin-Fetch are Wrenda fetching your origin.
  if (WRENDA_BOT_PATTERN.test(userAgent) && !req.headers['x-wrenda-origin-fetch']) {
    return wrendaProxy(req, res, next);
  }
  next();
});

// Your existing routes below...

HAProxy

haproxy.cfg
# Wrenda AI Crawler Proxy Configuration for HAProxy
# Add this to your HAProxy configuration.
#
# Forwards ALL bot traffic to Wrenda. Per-bot routing decisions happen
# server-side via routing templates in your dashboard.
#
# Replace YOUR_WRENDA_API_KEY with an API key from
# Settings → API & MCP (a read-scoped key is enough).

frontend http_front
    bind *:80
    bind *:443 ssl crt /path/to/cert.pem

    # Detect bots via the canonical Wrenda pattern
    acl is_wrenda_bot hdr_reg(User-Agent) -i GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot
    # Wrenda fetching your origin — never route it back to Wrenda
    acl is_wrenda_fetch req.hdr(X-Wrenda-Origin-Fetch) -m found

    # Forward bot traffic to Wrenda
    use_backend wrenda_backend if is_wrenda_bot !is_wrenda_fetch

    # Default backend for non-bot traffic
    default_backend origin_backend

backend wrenda_backend
    # X-Forwarded-Host must be captured before Host is rewritten
    http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    http-request set-header Host api.wrenda.ai
    http-request set-header Authorization "Bearer YOUR_WRENDA_API_KEY"
    http-request set-path /proxy/example.com%[path]
    server wrenda api.wrenda.ai:443 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt sni str(api.wrenda.ai)

backend origin_backend
    # Your origin server configuration
    server origin your-origin:80

CDN providers

For a CDN, Wrenda becomes an additional origin used only for bot traffic. Whatever the provider: send Host: api.wrenda.ai to that origin (not the viewer's Host), and don't route requests that carry X-Wrenda-Origin-Fetch to it — or set the domain's origin in Wrenda to a hostname that bypasses the CDN.

Cloudflare (zone stays proxied)

  1. Workers & Pages → Create → Hello World; replace the code with the Cloudflare Worker snippet above and deploy.
  2. In the zone: Workers Routes → Add route with the pattern your-domain.com/* and select the worker. Routes bind a pattern to a script; there is no separate origin field — the script decides where to fetch from.
  3. Verify with the curl below.

AWS CloudFront

Origin domain
api.wrenda.ai
Origin path
/proxy/your-domain.com
Protocol
HTTPS only
Custom origin header
Authorization = Bearer YOUR_WRENDA_API_KEY
Behaviour
Route bot user agents to this origin (an origin request policy that forwards User-Agent but not the viewer Host, or a CloudFront Function that rewrites the request).

Fastly

Add api.wrenda.ai as a backend with the override host and SNI hostname set to api.wrenda.ai, then prepend the proxy path for bot requests:

VCL snippet (vcl_miss / vcl_pass)
set bereq.url = "/proxy/your-domain.com" + bereq.url;
set bereq.http.Authorization = "Bearer YOUR_WRENDA_API_KEY";

KeyCDN

Origin URL
https://api.wrenda.ai/proxy/your-domain.com
SSL
Full (strict)
Forward Host header
Disabled (Wrenda needs Host: api.wrenda.ai)
Origin request header
Authorization: Bearer YOUR_WRENDA_API_KEY

Any other CDN

Configure a second origin at api.wrenda.ai with the path prefix /proxy/your-domain.com, send Host: api.wrenda.ai, forward the User-Agent header, add Authorization: Bearer YOUR_WRENDA_API_KEY as an origin request header, use TLS to the origin, and route only bot user agents to it. If your CDN can't route by user agent, contact support with the provider name and your domain.

Verify

Wrenda cannot see your server's configuration, so it can't confirm the forwarding for you. After you click Save & Verify with Crawl Tester the domain is marked configured and you land in Bot Testing pre-loaded with GPTBot/1.0 and your domain.

bash
curl -I -H "User-Agent: GPTBot/1.0" "https://your-domain.com/"
curl -I -H "User-Agent: Googlebot/2.1" "https://your-domain.com/"
Expected result
  • X-Correlation-ID — the request was processed by Wrenda.
  • X-Crawler-Action: optimize (or pre-render) — a crawler rule matched.
  • X-AI-Optimized: true for GPTBot in Active mode; X-Pre-Rendered: true for Googlebot when a pre-render rule exists.
  • In Shadow mode, X-Wrenda-Mode: shadow and the original content.
  • The first crawler visit to a page Wrenda hasn't optimized yet gets your original page straight away, with X-Wrenda-Rewrite: queued and X-AI-Optimized: false. The rewrite runs in the background (within about five minutes) and later visits get the optimized copy. Run cache warming to have pages ready before crawlers arrive.

If none of these headers appear, your server isn't forwarding crawler traffic to Wrenda yet. A 401 means the Authorization header is missing or wrong.

Next

Bot handlingActions, bot categories, recommended rules, priority and testing.Serving modesActive, Shadow and Paused — what each does and when to use it.Cache & purgeTTLs, what counts as a request, purge hooks, sitemap watch and warming.APIAPI keys and JWT auth, the purge hook, the status widget and the /proxy endpoint.

Was this helpful?

Missing a step, or found something that doesn't match what you see? Tell us and we'll fix the page.

Contact us