AI Crawlers and robots.txt: Allow or Block GPTBot, ClaudeBot and PerplexityBot
Which AI crawler user agents exist, what each one does, and a robots.txt template that lets AI search cite you while opting out of model training if you want to.
Which AI crawlers should I know about?
The main ones are OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic's ClaudeBot; PerplexityBot; Common Crawl's CCBot; and two opt-out tokens, Google-Extended and Applebot-Extended. Each AI company separates crawling for model training from crawling for search answers, so you can allow one and block the other.
| User agent | Operator | Purpose | Typical choice |
|---|---|---|---|
GPTBot | OpenAI | Model training | Block to opt out of training |
OAI-SearchBot | OpenAI | ChatGPT search results | Allow to be cited in ChatGPT search |
ChatGPT-User | OpenAI | Pages a ChatGPT user asks it to open | Allow |
ClaudeBot | Anthropic | Model training | Block to opt out of training |
Claude-SearchBot / Claude-User | Anthropic | Search and user-requested fetches | Allow to be cited by Claude |
PerplexityBot | Perplexity | Perplexity search index | Allow to be cited by Perplexity |
Google-Extended | Token: Gemini training and grounding | Optional; does not affect Search | |
Applebot-Extended | Apple | Token: Apple model training | Optional; Applebot still indexes for Siri and Spotlight |
CCBot | Common Crawl | Open crawl used in many AI datasets | Block to opt out of dataset reuse |
User agents and their purposes change; check each operator's crawler documentation before relying on a rule for legal or contractual reasons.
Does blocking AI crawlers remove me from AI answers?
Blocking the search and user-request agents (OAI-SearchBot, ChatGPT-User, PerplexityBot) makes it much less likely that those assistants can retrieve and cite your pages. Blocking only training agents such as GPTBot, CCBot or Google-Extended stops future model training on your content but does not remove you from live AI search results.
Does Google-Extended affect Google Search or AI Overviews?
No. Google-Extended is a robots.txt control token, not a separate crawler. It controls whether content Googlebot already crawls may be used to train and ground Gemini models. Google Search, including AI Overviews, uses regular Googlebot, so blocking Google-Extended does not change your Search rankings.
How do I allow AI search but block AI training?
Add separate robots.txt groups: Allow / for OAI-SearchBot, ChatGPT-User and PerplexityBot, and Disallow / for GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended. Keep your normal User-agent: * rules for search engines. Rules take effect when each crawler next fetches robots.txt, usually within a day.
Copy-paste robots.txt template
# Search engines and AI search: allowed
User-agent: *
Allow: /
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Model training: opted out (delete this group to allow training)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://example.com/sitemap.xmlA crawler follows only the most specific group that names it, so an agent listed in its own group ignores the User-agent: * rules. Repeat any Disallow lines (for example /admin/) inside each named group.
How can I check which AI crawlers my site blocks?
Run a free SeoGauge page check. It reads your robots.txt and tests seven AI user agents (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot) against the exact page you scan, including wildcard and longest-match rules, and reports any that are blocked.
Crawler access is one of seven GEO checks. Pair it with an llms.txt file and the other fixes in the GEO guide; the weights are in the scoring methodology.
See how your site scores on SEO, AEO and GEO.
Measure your website — free