- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and more
The robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNS
Confirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccess
Count visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers compared
What GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.
- ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers compared
What ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.
- Block AI Training with robots.txt and Stay Visible in ChatGPT Search, Perplexity and Claude
An annotated robots.txt that blocks GPTBot, ClaudeBot or CCBot while staying in ChatGPT Search, Perplexity and Claude, plus the RFC 9309 pitfalls to avoid.
- AI Crawler User Agents List 2026 - GPTBot, ClaudeBot, PerplexityBot and robots.txt Tokens
GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and more, with operator, robots.txt token, purpose, compliance and official IP ranges.