IABIGFR · Français
Weekly digest

AI policy changes, week 45 of 2026 (November 2, 2026 – November 8, 2026)

4 change(s) on 4 top domains: 1 newly blocking AI crawlers, 2 unblocking, 1 new llms.txt.

Crawler by crawler

CrawlerNewly blockedUnblockedPartially restrictedRestrictions lifted
Perplexity-User Perplexity12
Amazonbot Amazon2
Applebot-Extended Apple2
Bytespider ByteDance2
CCBot Common Crawl11
ChatGPT-User OpenAI2
Claude-SearchBot Anthropic2
Claude-User Anthropic2
ClaudeBot Anthropic2
GPTBot OpenAI2
Google-Extended Google2
Googlebot Google2
OAI-SearchBot OpenAI2
PerplexityBot Perplexity2
meta-externalagent Meta2
Applebot Apple1
Bingbot Microsoft1
DuckAssistBot DuckDuckGo1
MistralAI-User Mistral AI1
meta-externalfetcher Meta1

All changes this week

  • unblockpartialmeraki.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot; lifted partial rules for Googlebot, Bingbot, Applebot, CCBot.
    Tranco rank 176 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,75 +0,0 @@
    -User-agent: 360spider
    -User-agent: 360spider-image
    -User-agent: 360spider-video
    -User-agent: adsbot-google
    -User-agent: adsbot-google-mobile
    -User-agent: adsbot-google-mobile-apps
    -User-agent: adidxbot
    -User-agent: applebot
    -User-agent: applenewsbot
    -User-agent: baiduspider
    -User-agent: baiduspider-image
    -User-agent: baiduspider-news
    -User-agent: baiduspider-video
    -User-agent: bingbot
    -User-agent: bingpreview
    -User-agent: bublupbot
    -User-agent: ccbot
    -User-agent: cliqzbot
    -User-agent: coccoc
    -User-agent: coccocbot-image
    -User-agent: coccocbot-web
    -User-agent: daumoa
    -User-agent: dazoobot
    -User-agent: deusu
    -User-agent: duckduckbot
    -User-agent: duckduckgo-favicons-bot
    -User-agent: euripbot
    -User-agent: exploratodo
    -User-agent: facebot
    -User-agent: feedly
    -User-agent: findxbot
    -User-agent: googlebot
    -User-agent: googlebot-image
    -User-agent: googlebot-mobile
    -User-agent: googlebot-news
    -User-agent: googlebot-video
    -User-agent: haosouspider
    -User-agent: ichiro
    -User-agent: istellabot
    …
  • unblockpartialcnbc.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; lifted partial rules for Googlebot.
    Tranco rank 486 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,77 +0,0 @@
    -User-agent: googlebot
    -Disallow: /*native-android-mobile
    -Disallow: /*native-android-tablet
    -Disallow: /*mobile-native
    -Disallow: /preview/
    -Disallow: /undefined/
    -Disallow: /proplayer
    -Disallow: /appchart/*
    -Disallow: /search/*
    -Disallow: /zephr/*
    -Disallow: /blaize/*
    -
    -User-agent: gptbot
    -User-agent: perplexitybot
    -User-agent: amazonbot
    -User-agent: anthropic-ai
    -User-agent: awariorssbot
    -User-agent: awariosmartbot
    -User-agent: bytespider
    -User-agent: ccbot
    -User-agent: huggingfacebot
    -User-agent: chatgpt-user
    -User-agent: claudebot
    -User-agent: claude-web
    -User-agent: dataforseobot
    -User-agent: diffbot
    -User-agent: facebookbot
    -User-agent: google-extended
    -User-agent: google-notebooklm
    -User-agent: google-agent
    -User-agent: magpie-crawler
    -User-agent: newsnow
    -User-agent: news-please
    -User-agent: omgili
    -User-agent: omgilibot
    -User-agent: peer39_crawler
    -User-agent: peer39_crawler/1.0
    -User-agent: scrapy
    -User-agent: turnitinbot
    …
  • llms.txttarget.com published an llms.txt.
    Tranco rank 608 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,81 @@
    +User-agent: *
    +Disallow: /*/Ntk
    +Disallow: /*/Ntt
    +Disallow: /*/Ntx
    +Disallow: /*%7Cd_
    +Disallow: /*/schoollist/
    +Disallow: /*BTWN
    +Disallow: /[path]/
    +Disallow: /7078046/
    +Disallow: /7079046/
    +Disallow: /AddToList
    +Disallow: /AddToRegistry
    +Disallow: /admin
    +Disallow: /advancedGiftRegistrySearchView
    +Disallow: /AjaxSearchNavigationView
    +Disallow: /Allons_voter
    +Disallow: /bp/c/
    +Disallow: /bp/guest_mfg_brand
    +Disallow: /bp/p/
    +Disallow: /CallToActionModalView
    +Disallow: /cgi-bin
    +Disallow: /cgi-local
    +Disallow: /Checkout
    +Disallow: /CheckoutEditItemsDisplayView
    +Disallow: /CheckoutOrderBillingView
    +Disallow: /CheckoutOrderShippingView
    +Disallow: /CheckoutSignInView
    +Disallow: /co-
    +Disallow: /common
    +Disallow: /coupons.
    +Disallow: /custom-reviews/
    +Disallow: /data
    +Disallow: /database/philboard.mdb
    +Disallow: /deals/
    +Disallow: /dir_on_server/
    +Disallow: /EmailCartView
    +Disallow: /EnlargedImageView
    +Disallow: /ESPDisplayOptionsViewCmd
    +Disallow: /ESPModal
    …
  • blockdw.com blocked Perplexity-User.
    Tranco rank 1,390 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -63 +63,4 @@
     Disallow: /
    +
    +User-agent: perplexity-user
    +Disallow: /

All changes · Next week →

robots.txt and llms.txt of the top 2,000 Tranco domains, checked twice a day; a change is listed once it has been seen in two consecutive checks. How it works.