Attrape-Robots

Block or allow AI crawlers in robots.txt

Decide by purpose (training, indexing, on-demand fetch) and write the matching robots.txt lines.

  1. Choose by purpose: blocking training costs no visibility; blocking indexing or on-demand fetch removes you from answers.
  2. Write one rule per bot as below, or tick your choices in the generator.
  3. Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and ChatGPT-User allowed if you want to stay cited.
  4. robots.txt is a request: against bots that ignore it, filter by IP (verify a bot’s IP).
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

See also