Crawler by crawler
| Crawler | Newly blocked | Unblocked | Partially restricted | Restrictions lifted |
|---|---|---|---|---|
| CCBot Common Crawl | 1 | 1 | ||
| ChatGPT-User OpenAI | 1 | 1 | ||
| GPTBot OpenAI | 1 | 1 | ||
| Googlebot Google | 1 | 1 | ||
| Amazonbot Amazon | 1 | |||
| Applebot Apple | 1 | |||
| Applebot-Extended Apple | 1 | |||
| Bytespider ByteDance | 1 | |||
| Claude-SearchBot Anthropic | 1 | |||
| Claude-User Anthropic | 1 | |||
| ClaudeBot Anthropic | 1 | |||
| DuckAssistBot DuckDuckGo | 1 | |||
| Google-Extended Google | 1 | |||
| MistralAI-User Mistral AI | 1 | |||
| OAI-SearchBot OpenAI | 1 | |||
| Perplexity-User Perplexity | 1 | |||
| PerplexityBot Perplexity | 1 | |||
| meta-externalagent Meta | 1 | |||
| meta-externalfetcher Meta | 1 |
All changes this week
- blockpartialarchiveofourown.org blocked GPTBot, ChatGPT-User, CCBot; partially restricted Googlebot.
Tranco rank 1,073 · November 16, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,33 @@ +User-agent: * +Disallow: /works? +Disallow: /autocomplete/ +Disallow: /downloads/ +Disallow: /external_works/ +Disallow: /bookmarks/search? +Disallow: /people/search? +Disallow: /tags/search? +Disallow: /works/search? + +User-agent: googlebot +Disallow: /autocomplete/ +Disallow: /downloads/ +Disallow: /external_works/ +Disallow: /works/*? +Disallow: /*search? +Disallow: /*?*query= +Disallow: /*?*sort_ +Disallow: /*?*selected_tags +Disallow: /*?*view_adult +Disallow: /*?*tag_id +Disallow: /*?*pseud_id +Disallow: /*?*user_id +Disallow: /*?*pseud= + +User-agent: ccbot +Disallow: / + +User-agent: gptbot +Disallow: / + +User-agent: chatgpt-user +Disallow: /
- unblockfuraffinity.net unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Tranco rank 1,228 · November 16, 2026robots.txt diff (AI-relevant groups)
@@ -1,81 +0,0 @@ -User-agent: addsearchbot -User-agent: agenttimes -User-agent: ai2bot -User-agent: ai2bot-deepresearcheval -User-agent: ai2bot-dolma -User-agent: aihitbot -User-agent: amazon-kendra -User-agent: amazonbot -User-agent: amazonbuyforme -User-agent: amzn-searchbot -User-agent: amzn-user -User-agent: andibot -User-agent: anomura -User-agent: anthropic-ai -User-agent: apifybot -User-agent: apifywebsitecontentcrawler -User-agent: applebot -User-agent: applebot-extended -User-agent: aranet-searchbot -User-agent: atlassian-bot -User-agent: awario -User-agent: azureai-searchbot -User-agent: bedrockbot -User-agent: bigsur.ai -User-agent: bravebot -User-agent: brightbot -User-agent: brightbot 1.0 -User-agent: buddybot -User-agent: bytespider -User-agent: ccbot -User-agent: channel3bot -User-agent: chatglm-spider -User-agent: chatgpt agent -User-agent: chatgpt-user -User-agent: claude-code -User-agent: claude-searchbot -User-agent: claude-user -User-agent: claude-web -User-agent: claudebot …
- partialllms.txtpewresearch.org lifted partial rules for Googlebot; published an llms.txt.
Tranco rank 1,852 · November 16, 2026robots.txt diff (AI-relevant groups)
@@ -1,2 +1,5 @@ User-agent: * +Disallow: /search/ +Disallow: /search +Disallow: /?s= Disallow: /wp-admin/ @@ -4,8 +7,2 @@ Disallow: /wp-content/plugins/prc-icon-library/ - -User-agent: googlebot -Disallow: /wp-content/plugins/prc-icon-library/ -Disallow: /search/ -Disallow: /search -Disallow: /?s= Allow: /*/topline
robots.txt and llms.txt of the top 2,000 Tranco domains, checked twice a day; a change is listed once it has been seen in two consecutive checks. How it works.