Latest files: /blocklists/ai-bots.json
Choose a policy
| File | Block AI training only | Block all AI crawlers |
|---|---|---|
| robots.txt rules Polite request, respected by the major crawlers | robots-ai-training.txt | robots-ai-all.txt |
| nginx User-Agent block (403) | nginx-ai-training.conf | nginx-ai-all.conf |
| Apache .htaccess | htaccess-ai-training.txt | htaccess-ai-all.txt |
| Cloudflare WAF expression | cloudflare-ai-training.txt | cloudflare-ai-all.txt |
| Official IP ranges (one CIDR per line) | ips-ai-training.txt | ips-ai-all.txt |
| nginx IP deny list | nginx-deny-ai-training.conf | nginx-deny-ai-all.conf |
- Block AI training only: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot and Bytespider. Your pages can still be found and cited by ChatGPT Search, Claude, Perplexity and other AI search engines.
- Block all AI crawlers: the training crawlers plus AI search crawlers and the agents that fetch pages when a user asks (ChatGPT-User, Claude-User, Perplexity-User…). Your site disappears from AI answers.
- In both cases Googlebot, Bingbot and Applebot are never blocked: blocking them would remove you from Google, Bing and Apple search. Their AI use is controlled by the Google-Extended and Applebot-Extended robots.txt tokens, which the files include.
Which file should I use?
- Start with robots.txt. It is how the companies themselves ask you to opt out, and the major crawlers document that they respect it. Paste the rules into your existing robots.txt.
- Add a User-Agent block (nginx, Apache or Cloudflare) if you want a hard refusal instead of a request. It returns 403 to any request that announces itself as one of these crawlers.
- Use the IP lists to block at the firewall, or to check that a visitor claiming to be GPTBot really is (see verify a crawler IP). They contain only ranges published by OpenAI, Anthropic, Perplexity, Mistral AI, Common Crawl and DuckDuckGo.
Limits to know
- robots.txt is a request, not a lock. A crawler that ignores it is better handled by the User-Agent and IP files.
- A User-Agent can be faked; IP ranges cannot. But only some companies publish ranges, so a crawler without an official list is only blocked by name.
- Blocking has no effect on content already collected. It applies from the moment the crawler next visits.
Fetch the files automatically (for example once a day with curl), or use the machine-readable ai-bots.json. See every crawler's profile in the AI crawler directory, or build your own rules with the robots.txt generator.
More free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Verify an AI crawler IP
Check whether an IP address really belongs to OpenAI, Anthropic, Perplexity, Google, Bing, Apple or another AI company using their official IP lists.
Free llms.txt generator from your sitemap
Generate a draft llms.txt in the llmstxt.org format from your website's sitemap - titles, descriptions and sections ready to review.
Free prompt injection detector
Check whether a web page or a text hides instructions aimed at AI assistants - hidden text, HTML comments and invisible Unicode characters.
AI API cost calculator
Compare what 40 AI models (OpenAI, Anthropic, Google, Mistral, DeepSeek, xAI) really cost for your volumes: verified official prices, caching, batch and long-context tiers.