IABIGFR · Français
Weekly digest

AI policy changes, week 46 of 2026 (November 9, 2026 – November 15, 2026)

19 change(s) on 16 top domains: 5 newly blocking AI crawlers, 10 unblocking, 1 new llms.txt.

Crawler by crawler

CrawlerNewly blockedUnblockedPartially restrictedRestrictions lifted
ClaudeBot Anthropic44
Bytespider ByteDance43
GPTBot OpenAI241
Applebot-Extended Apple321
CCBot Common Crawl24
ChatGPT-User OpenAI321
Google-Extended Google231
Perplexity-User Perplexity321
PerplexityBot Perplexity321
Amazonbot Amazon23
Claude-SearchBot Anthropic311
Claude-User Anthropic311
OAI-SearchBot OpenAI311
meta-externalagent Meta32
DuckAssistBot DuckDuckGo211
MistralAI-User Mistral AI211
Googlebot Google111
meta-externalfetcher Meta21
Applebot Apple11
Bingbot Microsoft11

All changes this week

  • unblockjustdial.com unblocked ClaudeBot.
    Tranco rank 1,341 · November 9, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,56 +0,0 @@
    -User-agent: *
    -Disallow: /functions/
    -Disallow: /webmain/
    -Disallow: /*/wcdata/
    -Disallow: /_next/
    -Disallow: /web-worker/
    -Disallow: /us/
    -Disallow: /ca/
    -Disallow: /jd-trkr
    -Disallow: /api/
    -Disallow: /jdmart/api/
    -Disallow: /real-estate/services/api/
    -Disallow: /api/php/appUserId
    -Disallow: /gapi/ratings
    -Disallow: /api/india_api_write
    -Disallow: /api/india_api_read
    -Disallow: /bestdeals_chatbot/
    -
    -User-agent: oai-searchbot
    -Allow: /
    -
    -User-agent: chatgpt-user
    -Allow: /
    -
    -User-agent: claude-user
    -Allow: /
    -
    -User-agent: claude-searchbot
    -Allow: /
    -
    -User-agent: claudebot
    -Disallow: /
    -
    -User-agent: perplexitybot
    -Allow: /
    -
    -User-agent: perplexity-user
    -Allow: /
    -
    …
  • unblockladies.de unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
    Tranco rank 1,562 · November 9, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,2 +1,28 @@
     User-agent: *
    -Disallow: /
    +Allow: /
    +Disallow: /*?*umkreis=*adresse=*
    +Disallow: /*?*suchtext=*adresse=*
    +Disallow: /*?*umkreis=70*
    +Disallow: /*?*umkreis=100*
    +Disallow: /*?*umkreis=150*
    +Disallow: /*?*umkreis=200*
    +Disallow: /*?*umkreis=300*
    +Disallow: /*?*umkreis=70$
    +Disallow: /*?*umkreis=100$
    +Disallow: /*?*umkreis=150$
    +Disallow: /*?*umkreis=200$
    +Disallow: /*?*umkreis=300$
    +Disallow: /*?page=1$
    +Disallow: /*?*page=1$
    +Disallow: /*?*page=1&*
    +Disallow: /*?*&name=*
    +Disallow: /sex-anzeigen*umkreis=*
    +Disallow: /cdn-cgi
    +Disallow: /login
    +Disallow: /api/affiliates
    +Disallow: /api/marketplace-banners
    +Disallow: /api/bookmarks
    +Disallow: /api/affiliates*
    +Disallow: /api/marketplace-banners*
    +Disallow: /api/bookmarks*
    +Disallow: /*%3D*
  • blockpartialcnbc.com blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; partially restricted Googlebot.
    Tranco rank 486 · November 10, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,77 @@
    +User-agent: googlebot
    +Disallow: /*native-android-mobile
    +Disallow: /*native-android-tablet
    +Disallow: /*mobile-native
    +Disallow: /preview/
    +Disallow: /undefined/
    +Disallow: /proplayer
    +Disallow: /appchart/*
    +Disallow: /search/*
    +Disallow: /zephr/*
    +Disallow: /blaize/*
    +
    +User-agent: gptbot
    +User-agent: perplexitybot
    +User-agent: amazonbot
    +User-agent: anthropic-ai
    +User-agent: awariorssbot
    +User-agent: awariosmartbot
    +User-agent: bytespider
    +User-agent: ccbot
    +User-agent: huggingfacebot
    +User-agent: chatgpt-user
    +User-agent: claudebot
    +User-agent: claude-web
    +User-agent: dataforseobot
    +User-agent: diffbot
    +User-agent: facebookbot
    +User-agent: google-extended
    +User-agent: google-notebooklm
    +User-agent: google-agent
    +User-agent: magpie-crawler
    +User-agent: newsnow
    +User-agent: news-please
    +User-agent: omgili
    +User-agent: omgilibot
    +User-agent: peer39_crawler
    +User-agent: peer39_crawler/1.0
    +User-agent: scrapy
    +User-agent: turnitinbot
    …
  • unblockcorriere.it unblocked Amazonbot.
    Tranco rank 959 · November 10, 2026
    robots.txt diff (AI-relevant groups)
    @@ -20,3 +20,3 @@
     User-agent: amazonbot
    -Disallow: /
    +Allow: /
     
  • partialllms.txtiporntv.net partially restricted GPTBot; published an llms.txt.
    Tranco rank 1,527 · November 10, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,15 @@
    +User-agent: *
    +Disallow: /player/float.php
    +Disallow: /player/html.php
    +Disallow: /*?action=*$
    +Disallow: /*mode=async*$
    +Disallow: /*function=get_block*$
    +Disallow: /404.php
    +Disallow: /captcha/*
    +Disallow: /link/*
    +Disallow: /related_videos_html/*
    +Disallow: /feedback/
    +
    +User-agent: gptbot
    +Disallow: /s/
    +Disallow: /get_file/
  • partialfiverr.com partially restricted OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, MistralAI-User, DuckAssistBot.
    Tranco rank 828 · November 11, 2026
    robots.txt diff (AI-relevant groups)
    @@ -56 +56,26 @@
     Disallow: /logo-maker/woo
    +
    +User-agent: google-extended
    +User-agent: oai-searchbot
    +User-agent: claude-searchbot
    +User-agent: perplexitybot
    +User-agent: mistralai-index
    +User-agent: applebot-extended
    +User-agent: youbot
    +Allow: /
    +Disallow: /orders/timeline/*
    +Disallow: */pinned_flashes/*
    +Disallow: /gigs/*/share/
    +Disallow: /gigs/*/share?*
    +Disallow: /specials/*
    +Disallow: /packages/*
    +Disallow: /categories/silly
    +Disallow: /categories/fifa
    +Disallow: /categories/Halloween
    +Disallow: /categories/Postcards
    +Disallow: /purchases
    +Disallow: /user_sessions
    +Disallow: /users/
    +Disallow: /counter/*?
    +Disallow: /collaborate/*
    +…
  • blockgazzetta.it blocked OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, Bytespider, DuckAssistBot.
    Tranco rank 1,237 · November 12, 2026
    robots.txt diff (AI-relevant groups)
    @@ -27 +27,28 @@
     Disallow: /
    +
    +User-agent: bytespider
    +Disallow: /
    +
    +User-agent: chatgpt-user
    +Disallow: /
    +
    +User-agent: oai-searchbot
    +Disallow: /
    +
    +User-agent: claude-user
    +Disallow: /
    +
    +User-agent: claude-searchbot
    +Disallow: /
    +
    +User-agent: perplexitybot
    +Disallow: /
    +
    +User-agent: perplexity-user
    +Disallow: /
    +
    +User-agent: duckassistbot
    +Disallow: /
    +
    +User-agent: mistralai-user
    +Disallow: /
  • blockjustdial.com blocked ClaudeBot.
    Tranco rank 1,341 · November 12, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,56 @@
    +User-agent: *
    +Disallow: /functions/
    +Disallow: /webmain/
    +Disallow: /*/wcdata/
    +Disallow: /_next/
    +Disallow: /web-worker/
    +Disallow: /us/
    +Disallow: /ca/
    +Disallow: /jd-trkr
    +Disallow: /api/
    +Disallow: /jdmart/api/
    +Disallow: /real-estate/services/api/
    +Disallow: /api/php/appUserId
    +Disallow: /gapi/ratings
    +Disallow: /api/india_api_write
    +Disallow: /api/india_api_read
    +Disallow: /bestdeals_chatbot/
    +
    +User-agent: oai-searchbot
    +Allow: /
    +
    +User-agent: chatgpt-user
    +Allow: /
    +
    +User-agent: claude-user
    +Allow: /
    +
    +User-agent: claude-searchbot
    +Allow: /
    +
    +User-agent: claudebot
    +Disallow: /
    +
    +User-agent: perplexitybot
    +Allow: /
    +
    +User-agent: perplexity-user
    +Allow: /
    +
    …
  • blockladies.de blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
    Tranco rank 1,562 · November 12, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,28 +1,2 @@
     User-agent: *
    -Allow: /
    -Disallow: /*?*umkreis=*adresse=*
    -Disallow: /*?*suchtext=*adresse=*
    -Disallow: /*?*umkreis=70*
    -Disallow: /*?*umkreis=100*
    -Disallow: /*?*umkreis=150*
    -Disallow: /*?*umkreis=200*
    -Disallow: /*?*umkreis=300*
    -Disallow: /*?*umkreis=70$
    -Disallow: /*?*umkreis=100$
    -Disallow: /*?*umkreis=150$
    -Disallow: /*?*umkreis=200$
    -Disallow: /*?*umkreis=300$
    -Disallow: /*?page=1$
    -Disallow: /*?*page=1$
    -Disallow: /*?*page=1&*
    -Disallow: /*?*&name=*
    -Disallow: /sex-anzeigen*umkreis=*
    -Disallow: /cdn-cgi
    -Disallow: /login
    -Disallow: /api/affiliates
    -Disallow: /api/marketplace-banners
    -Disallow: /api/bookmarks
    -Disallow: /api/affiliates*
    -Disallow: /api/marketplace-banners*
    -Disallow: /api/bookmarks*
    -Disallow: /*%3D*
    +Disallow: /
  • unblocktechcrunch.com unblocked ChatGPT-User, Google-Extended.
    Tranco rank 633 · November 13, 2026
    robots.txt diff (AI-relevant groups)
    @@ -17,10 +17,3 @@
     
    -User-agent: chatgpt-user
    -Disallow: /
    -
    -User-agent: claude-web
     User-agent: claudebot
    -Disallow: /
    -
    -User-agent: google-extended
     Disallow: /
  • blockhbr.org blocked ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider.
    Tranco rank 912 · November 13, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,2 +1,3 @@
     User-agent: *
    +Allow: /sponsored/
     Disallow: /resources/
    @@ -37 +38,16 @@
     Disallow: /
    +
    +User-agent: claudebot
    +Disallow: /
    +
    +User-agent: applebot-extended
    +Disallow: /
    +
    +User-agent: bytespider
    +Disallow: /
    +
    +User-agent: meta-externalagent
    +Disallow: /
    +
    +User-agent: meta-externalfetcher
    +Disallow: /
  • unblockhola.com unblocked CCBot.
    Tranco rank 1,311 · November 13, 2026
  • llms.txtpewresearch.org removed its llms.txt.
    Tranco rank 1,846 · November 13, 2026
  • unblockpbs.org unblocked GPTBot, PerplexityBot, CCBot, Bytespider.
    Tranco rank 970 · November 14, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,81 +0,0 @@
    -User-agent: *
    -Disallow: /wgbh/pages/frontline/digitalnation/%3F
    -Disallow: /wgbh/pages/frontline/digitalnation/%3F*
    -Disallow: /wgbh/frontline/*elqat*
    -Disallow: /wgbh/pages/frontline/*elqat*
    -Disallow: /wgbh/pages/frontline/bushswar*
    -Disallow: /independentlens/getinvolved/cinema/
    -Disallow: /cgi-bin/*
    -Disallow: /cgi-registry/*
    -Disallow: /survey/
    -Disallow: /whatson/stations/
    -Disallow: /whatson/schedulex/
    -Disallow: /whatson/pbsyou/
    -Disallow: /whatson/dbs/
    -Disallow: /remotecontrol/bestpractices/prototype/
    -Disallow: /remotecontrol/bestpractices/jl-prototype/
    -Disallow: /pressroom/
    -Disallow: /als/shakespeare_pilot/
    -Disallow: /newshour/bb/military/military.html/
    -Disallow: /producers/webmanual/
    -Disallow: /devcon2007/
    -Disallow: /smcot/
    -Disallow: /pbsholidaycard/
    -Disallow: /gpra/
    -Disallow: /tlqa/
    -Disallow: /remote.control/
    -Disallow: /teachers/connect/
    -Disallow: /engage/sites/all/files/blog_images/monkeys.gif
    -Disallow: /teachers/edcar/
    -Disallow: /teachers/dll_research/
    -Disallow: /peerconnection/feeds/
    -Disallow: /teachers/innovators/toolkit/
    -Disallow: /mt4/
    -Disallow: /klru/mt4/
    -Disallow: /newshour/mt4/
    -Disallow: /mediashift/mt4/
    -Disallow: /kcet/mt4/
    -Disallow: /newshour/mt3/
    -Disallow: /now/mt3/
    …
  • unblockjustdial.com unblocked ClaudeBot.
    Tranco rank 1,341 · November 14, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,56 +0,0 @@
    -User-agent: *
    -Disallow: /functions/
    -Disallow: /webmain/
    -Disallow: /*/wcdata/
    -Disallow: /_next/
    -Disallow: /web-worker/
    -Disallow: /us/
    -Disallow: /ca/
    -Disallow: /jd-trkr
    -Disallow: /api/
    -Disallow: /jdmart/api/
    -Disallow: /real-estate/services/api/
    -Disallow: /api/php/appUserId
    -Disallow: /gapi/ratings
    -Disallow: /api/india_api_write
    -Disallow: /api/india_api_read
    -Disallow: /bestdeals_chatbot/
    -
    -User-agent: oai-searchbot
    -Allow: /
    -
    -User-agent: chatgpt-user
    -Allow: /
    -
    -User-agent: claude-user
    -Allow: /
    -
    -User-agent: claude-searchbot
    -Allow: /
    -
    -User-agent: claudebot
    -Disallow: /
    -
    -User-agent: perplexitybot
    -Allow: /
    -
    -User-agent: perplexity-user
    -Allow: /
    -
    …
  • llms.txtunesco.org removed its llms.txt.
    Tranco rank 614 · November 15, 2026
  • unblockdw.com unblocked Perplexity-User.
    Tranco rank 1,400 · November 15, 2026
    robots.txt diff (AI-relevant groups)
    @@ -63,4 +63 @@
     Disallow: /
    -
    -User-agent: perplexity-user
    -Disallow: /
  • unblockloteriadehoy.com unblocked GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider.
    Tranco rank 1,748 · November 15, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,28 +1 @@
     User-agent: *
    -Allow: /
    -
    -User-agent: amazonbot
    -Disallow: /
    -
    -User-agent: applebot-extended
    -Disallow: /
    -
    -User-agent: bytespider
    -Disallow: /
    -
    -User-agent: ccbot
    -Disallow: /
    -
    -User-agent: claudebot
    -Disallow: /
    -
    -User-agent: google-extended
    -Disallow: /
    -
    -User-agent: gptbot
    -Disallow: /
    -
    -User-agent: meta-externalagent
    -Disallow: /
    -
    -User-agent: *
  • unblocksattamatkano1.me unblocked GPTBot.
    Tranco rank 1,789 · November 15, 2026
    robots.txt diff (AI-relevant groups)
    @@ -5,4 +5 @@
     Allow: /
    -
    -User-agent: gptbot
    -Disallow: /

← Previous week · All changes · Next week →

robots.txt and llms.txt of the top 2,000 Tranco domains, checked twice a day; a change is listed once it has been seen in two consecutive checks. How it works.