Attrape-Robots

Verify an AI crawler’s IP address

Compare the origin address with official ranges: an agent name proves nothing.

  1. Get the visitor’s real address from your logs (behind a CDN, see the CDN guide).
  2. Compare it with the ranges the publisher publishes, listed in the crawler list.
  3. Without a published range, rely on the publisher’s documentation; Google and Bing also document reverse-DNS verification.
  4. Re-download the files regularly: publishers change ranges without notice.
curl -s https://openai.com/gptbot.json

See also