Attrape-Robots

AI crawlers behind a CDN: recover the real IP

Behind a CDN your logs show the proxy’s address. Here is how to recover the bot’s.

  1. Behind Cloudflare, Fastly or any reverse proxy, your logs record the CDN’s address, identical for a bot and a human.
  2. The real address travels in a header set by the proxy: trust it on your origin (nginx example below).
  3. set_real_ip_from is mandatory: without it anyone can forge the header and pose as Googlebot.
  4. Or filter in a CDN WAF rule, where the source address is already correct. Do not hard-code ranges in configuration.
real_ip_header CF-Connecting-IP;
set_real_ip_from ;

See also