IABIGFR · Français
Weekly digest

AI policy changes, week 47 of 2026 (November 16, 2026 – November 22, 2026)

3 change(s) on 3 top domains: 1 newly blocking AI crawlers, 1 unblocking, 1 new llms.txt.

Crawler by crawler

CrawlerNewly blockedUnblockedPartially restrictedRestrictions lifted
CCBot Common Crawl11
ChatGPT-User OpenAI11
GPTBot OpenAI11
Googlebot Google11
Amazonbot Amazon1
Applebot Apple1
Applebot-Extended Apple1
Bytespider ByteDance1
Claude-SearchBot Anthropic1
Claude-User Anthropic1
ClaudeBot Anthropic1
DuckAssistBot DuckDuckGo1
Google-Extended Google1
MistralAI-User Mistral AI1
OAI-SearchBot OpenAI1
Perplexity-User Perplexity1
PerplexityBot Perplexity1
meta-externalagent Meta1
meta-externalfetcher Meta1

All changes this week

  • blockpartialarchiveofourown.org blocked GPTBot, ChatGPT-User, CCBot; partially restricted Googlebot.
    Tranco rank 1,073 · November 16, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,33 @@
    +User-agent: *
    +Disallow: /works?
    +Disallow: /autocomplete/
    +Disallow: /downloads/
    +Disallow: /external_works/
    +Disallow: /bookmarks/search?
    +Disallow: /people/search?
    +Disallow: /tags/search?
    +Disallow: /works/search?
    +
    +User-agent: googlebot
    +Disallow: /autocomplete/
    +Disallow: /downloads/
    +Disallow: /external_works/
    +Disallow: /works/*?
    +Disallow: /*search?
    +Disallow: /*?*query=
    +Disallow: /*?*sort_
    +Disallow: /*?*selected_tags
    +Disallow: /*?*view_adult
    +Disallow: /*?*tag_id
    +Disallow: /*?*pseud_id
    +Disallow: /*?*user_id
    +Disallow: /*?*pseud=
    +
    +User-agent: ccbot
    +Disallow: /
    +
    +User-agent: gptbot
    +Disallow: /
    +
    +User-agent: chatgpt-user
    +Disallow: /
  • unblockfuraffinity.net unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
    Tranco rank 1,228 · November 16, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,81 +0,0 @@
    -User-agent: addsearchbot
    -User-agent: agenttimes
    -User-agent: ai2bot
    -User-agent: ai2bot-deepresearcheval
    -User-agent: ai2bot-dolma
    -User-agent: aihitbot
    -User-agent: amazon-kendra
    -User-agent: amazonbot
    -User-agent: amazonbuyforme
    -User-agent: amzn-searchbot
    -User-agent: amzn-user
    -User-agent: andibot
    -User-agent: anomura
    -User-agent: anthropic-ai
    -User-agent: apifybot
    -User-agent: apifywebsitecontentcrawler
    -User-agent: applebot
    -User-agent: applebot-extended
    -User-agent: aranet-searchbot
    -User-agent: atlassian-bot
    -User-agent: awario
    -User-agent: azureai-searchbot
    -User-agent: bedrockbot
    -User-agent: bigsur.ai
    -User-agent: bravebot
    -User-agent: brightbot
    -User-agent: brightbot 1.0
    -User-agent: buddybot
    -User-agent: bytespider
    -User-agent: ccbot
    -User-agent: channel3bot
    -User-agent: chatglm-spider
    -User-agent: chatgpt agent
    -User-agent: chatgpt-user
    -User-agent: claude-code
    -User-agent: claude-searchbot
    -User-agent: claude-user
    -User-agent: claude-web
    -User-agent: claudebot
    …
  • partialllms.txtpewresearch.org lifted partial rules for Googlebot; published an llms.txt.
    Tranco rank 1,852 · November 16, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,2 +1,5 @@
     User-agent: *
    +Disallow: /search/
    +Disallow: /search
    +Disallow: /?s=
     Disallow: /wp-admin/
    @@ -4,8 +7,2 @@
     Disallow: /wp-content/plugins/prc-icon-library/
    -
    -User-agent: googlebot
    -Disallow: /wp-content/plugins/prc-icon-library/
    -Disallow: /search/
    -Disallow: /search
    -Disallow: /?s=
     Allow: /*/topline

← Previous week · All changes

robots.txt and llms.txt of the top 2,000 Tranco domains, checked twice a day; a change is listed once it has been seen in two consecutive checks. How it works.