AI crawler checker: does your robots.txt block ChatGPT, Claude and Perplexity?
Enter a website and see which AI crawlers its robots.txt allows or blocks: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more. Then copy a ready-made policy.
Copy a ready-made AI crawler policy
Choose a policy and paste it into your robots.txt. Remove any older rules for the same crawlers so they don't conflict.
# AI crawlers: allowed to search and cite, not to train models
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: GoogleOther
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: MistralAI-User
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: cohere-ai
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
How do I check if my site blocks AI crawlers?
Enter your domain above. The checker reads your robots.txt the way crawlers do (longest matching rule wins, Allow wins ties) and shows, for each AI crawler, whether it may visit your homepage and whether your file names it directly or it falls under the User-agent: * rules.
Which AI crawlers should I allow?
To be found and cited by AI assistants, allow the search and live-fetch crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot and Perplexity-User. Training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended are a separate choice: blocking them keeps your content out of future model training but doesn't stop AI search from citing you.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and when someone asks ChatGPT to open a page it uses ChatGPT-User. Blocking only GPTBot keeps your content out of training while ChatGPT can still find and cite you.
My robots.txt allows AI crawlers but they still can't get in. Why?
robots.txt is a request, not a gate. Firewalls and bot-protection services (for example "block AI bots" settings at a CDN) can refuse AI crawlers before robots.txt matters. Use the free AI crawler log check to see whether the bots actually reach your server.
Read the full guide to AI crawlers and robots.txt →
See how your site scores on SEO, AEO and GEO.
Measure your website — free