Attrape-Robots

Frequently asked questions

Does blocking GPTBot remove me from ChatGPT answers?

No. GPTBot is for training. ChatGPT search uses OAI-SearchBot, and user-requested page visits use ChatGPT-User. Blocking OAI-SearchBot removes you from sourced answers.

Does a User-Agent prove a bot is genuine?

No. Anyone can write GPTBot in a header. Compare the origin IP with the ranges the publisher publishes.

What if my site is behind a CDN?

Your logs show the CDN’s address. Use the real-IP header set by the proxy (with a trusted proxy list) or filter in a CDN WAF rule.

Is robots.txt enough to block a bot?

No. Reputable bots obey it, others ignore it. Only IP filtering works against those.

AI crawlers · Diagnostic