Which AI Crawlers to Allow in robots.txt
Blocking an AI crawler is the most complete way to disappear from AI answers — and it is usually done by accident.
Every other problem on an audit is a matter of degree. This one is binary. If your robots.txt disallows a crawler from /, that company's model has no route to your content at all, no matter how good the content is.
The crawlers that matter
- GPTBot — OpenAI's training crawler. Content it collects can inform what ChatGPT knows about you.
- OAI-SearchBot — OpenAI's search crawler, powering live results in ChatGPT. Distinct from GPTBot; blocking one does not block the other.
- ChatGPT-User — fetches a page live when a user asks ChatGPT to look at it. Block this and you cannot even be pasted into a chat.
- ClaudeBot — Anthropic's crawler for Claude.
- PerplexityBot — Perplexity, which cites sources prominently and visibly.
- Google-Extended — controls Gemini and AI Overviews without affecting normal Google Search. Blocking it does not hurt your blue-link ranking, and allowing it does not change it either.
- CCBot — Common Crawl. Not an AI company itself, but its public dataset feeds a great many models. Easy to overlook and disproportionately costly to block.
- Applebot-Extended — Apple Intelligence.
How sites block them by accident
Almost nobody sets out to block AI. It happens three ways.
A blanket wildcard. A rule like User-agent: * with Disallow: /, left over from a staging environment, blocks everything including every crawler above.
A copied robots.txt. Boilerplate from a template or a plugin, often written when AI crawlers were seen purely as a threat, quietly carrying a dozen disallow rules the site owner never read.
A platform default. Some hosts and CMSs added AI-crawler blocks by default during the 2023–24 wave of concern. If you never edited robots.txt, you may be blocking crawlers you'd rather allow.
Read yours before changing it
Open https://yoursite.com/robots.txt. Look for any group whose user-agent matches a crawler above, and any Disallow: / under it or under User-agent: *. A crawler follows the most specific group that names it, falling back to * only when it has no group of its own.
Allowing them
An empty Disallow: means "nothing is disallowed" — the standard way to express full access:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /Place these before any wildcard group. If a path genuinely must stay private, disallow that path specifically rather than blocking the crawler outright.
Should you allow all of them?
That is a business decision, not a technical one, and it is reasonable to say no. If your revenue comes from people reading your pages, feeding a model that answers without sending a click is a real trade. Publishers and subscription sites weigh this differently from a business whose site exists to be found.
What is not defensible is blocking by accident. Decide deliberately, then make robots.txt say what you decided.
A note on severity: blocking one crawler is a partial loss. Blocking all of them is total. Our audit weights these differently for that reason — and it took a bug fix to get right, because an earlier version rated a blocked crawler as "low priority" on a tool built entirely around AI visibility.
robots.txt is not security
It is a request, honoured by well-behaved crawlers and ignored by everything else. Never use it to hide sensitive URLs — it is a public file that lists exactly which paths you consider worth hiding. Use authentication for anything that genuinely must not be read.
Run a free audit to see which AI crawlers your robots.txt currently allows.