Help › Fixing

robots.txt and AI crawlers

Which bots matter, and the one mistake that makes everything else pointless.

Last updated 2026-07-30

AI companies crawl the web with named bots. The ones that matter most today are GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and Google-Extended (Google's AI products).

Your robots.txt tells them whether they may read you. If it says no, they do not — and nothing else you do will matter, because you cannot be quoted from a page that was never fetched.

Check yours

Open yourdomain.com/robots.txt and look for Disallow rules under a User-agent line naming any of those bots. Many sites acquired these years ago, from a plugin default or a developer being cautious.

Blocking is a legitimate choice

Some businesses genuinely do not want their content used to train or ground AI answers. That is a defensible position. It is simply incompatible with wanting to appear in AI answers — you cannot have both, and it is better to decide deliberately than to discover the block by accident.

Related

• llms.txt, explained
• The schema markup worth adding
• What to do when the fix isn't on your website