AI crawlers behind a CDN: recover the real IP
Behind a CDN your logs show the proxy’s address. Here is how to recover the bot’s.
- Behind Cloudflare, Fastly or any reverse proxy, your logs record the CDN’s address, identical for a bot and a human.
- The real address travels in a header set by the proxy: trust it on your origin (nginx example below).
- set_real_ip_from is mandatory: without it anyone can forge the header and pose as Googlebot.
- Or filter in a CDN WAF rule, where the source address is already correct. Do not hard-code ranges in configuration.
real_ip_header CF-Connecting-IP;
set_real_ip_from ;