Block or allow AI crawlers in robots.txt
Decide by purpose (training, indexing, on-demand fetch) and write the matching robots.txt lines.
- Choose by purpose: blocking training costs no visibility; blocking indexing or on-demand fetch removes you from answers.
- Write one rule per bot as below, or tick your choices in the generator.
- Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User allowed if you want to stay cited.
- robots.txt is a request: against bots that ignore it, filter by IP (verify a bot’s IP).
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /