IABIGFR · Français
Checked twice a day

AI policy change tracker

Which of the world's most popular websites just blocked, or unblocked, GPTBot, ClaudeBot and the other AI crawlers? A dated feed of every change to the robots.txt and llms.txt of the top 2,000 domains.

Last check: November 16, 2026, 21:53 UTC · Atom feed

20AI policy changes in the last 7 days
26changes in the last 30 days
7sites that newly blocked AI crawlers (30 days)
13sites that unblocked AI crawlers (30 days)
3new llms.txt files (30 days)
1,493domains watched with a usable robots.txt

Crawler by crawler, last 30 days

Number of sites whose robots.txt changed for each crawler.

CrawlerNewly blockedUnblockedPartially restrictedRestrictions lifted
ClaudeBot Anthropic47
GPTBot OpenAI371
Bytespider ByteDance46
CCBot Common Crawl361
ChatGPT-User OpenAI451
Perplexity-User Perplexity451
Applebot-Extended Apple351
Google-Extended Google261
PerplexityBot Perplexity351
Amazonbot Amazon26
Claude-SearchBot Anthropic341
Claude-User Anthropic341
OAI-SearchBot OpenAI341
meta-externalagent Meta35
Googlebot Google1123
DuckAssistBot DuckDuckGo231
MistralAI-User Mistral AI231
meta-externalfetcher Meta23
Applebot Apple121
Bingbot Microsoft111

Latest changes

  • partialllms.txtpewresearch.org lifted partial rules for Googlebot; published an llms.txt.
    Tranco rank 1,852 · November 16, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,2 +1,5 @@
     User-agent: *
    +Disallow: /search/
    +Disallow: /search
    +Disallow: /?s=
     Disallow: /wp-admin/
    @@ -4,8 +7,2 @@
     Disallow: /wp-content/plugins/prc-icon-library/
    -
    -User-agent: googlebot
    -Disallow: /wp-content/plugins/prc-icon-library/
    -Disallow: /search/
    -Disallow: /search
    -Disallow: /?s=
     Allow: /*/topline
  • blockpartialarchiveofourown.org blocked GPTBot, ChatGPT-User, CCBot; partially restricted Googlebot.
    Tranco rank 1,073 · November 16, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,33 @@
    +User-agent: *
    +Disallow: /works?
    +Disallow: /autocomplete/
    +Disallow: /downloads/
    +Disallow: /external_works/
    +Disallow: /bookmarks/search?
    +Disallow: /people/search?
    +Disallow: /tags/search?
    +Disallow: /works/search?
    +
    +User-agent: googlebot
    +Disallow: /autocomplete/
    +Disallow: /downloads/
    +Disallow: /external_works/
    +Disallow: /works/*?
    +Disallow: /*search?
    +Disallow: /*?*query=
    +Disallow: /*?*sort_
    +Disallow: /*?*selected_tags
    +Disallow: /*?*view_adult
    +Disallow: /*?*tag_id
    +Disallow: /*?*pseud_id
    +Disallow: /*?*user_id
    +Disallow: /*?*pseud=
    +
    +User-agent: ccbot
    +Disallow: /
    +
    +User-agent: gptbot
    +Disallow: /
    +
    +User-agent: chatgpt-user
    +Disallow: /
  • unblockfuraffinity.net unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
    Tranco rank 1,228 · November 16, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,81 +0,0 @@
    -User-agent: addsearchbot
    -User-agent: agenttimes
    -User-agent: ai2bot
    -User-agent: ai2bot-deepresearcheval
    -User-agent: ai2bot-dolma
    -User-agent: aihitbot
    -User-agent: amazon-kendra
    -User-agent: amazonbot
    -User-agent: amazonbuyforme
    -User-agent: amzn-searchbot
    -User-agent: amzn-user
    -User-agent: andibot
    -User-agent: anomura
    -User-agent: anthropic-ai
    -User-agent: apifybot
    -User-agent: apifywebsitecontentcrawler
    -User-agent: applebot
    -User-agent: applebot-extended
    -User-agent: aranet-searchbot
    -User-agent: atlassian-bot
    -User-agent: awario
    -User-agent: azureai-searchbot
    -User-agent: bedrockbot
    -User-agent: bigsur.ai
    -User-agent: bravebot
    -User-agent: brightbot
    -User-agent: brightbot 1.0
    -User-agent: buddybot
    -User-agent: bytespider
    -User-agent: ccbot
    -User-agent: channel3bot
    -User-agent: chatglm-spider
    -User-agent: chatgpt agent
    -User-agent: chatgpt-user
    -User-agent: claude-code
    -User-agent: claude-searchbot
    -User-agent: claude-user
    -User-agent: claude-web
    -User-agent: claudebot
    …
  • unblocksattamatkano1.me unblocked GPTBot.
    Tranco rank 1,789 · November 15, 2026
    robots.txt diff (AI-relevant groups)
    @@ -5,4 +5 @@
     Allow: /
    -
    -User-agent: gptbot
    -Disallow: /
  • unblockdw.com unblocked Perplexity-User.
    Tranco rank 1,400 · November 15, 2026
    robots.txt diff (AI-relevant groups)
    @@ -63,4 +63 @@
     Disallow: /
    -
    -User-agent: perplexity-user
    -Disallow: /
  • llms.txtunesco.org removed its llms.txt.
    Tranco rank 614 · November 15, 2026
  • unblockloteriadehoy.com unblocked GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider.
    Tranco rank 1,748 · November 15, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,28 +1 @@
     User-agent: *
    -Allow: /
    -
    -User-agent: amazonbot
    -Disallow: /
    -
    -User-agent: applebot-extended
    -Disallow: /
    -
    -User-agent: bytespider
    -Disallow: /
    -
    -User-agent: ccbot
    -Disallow: /
    -
    -User-agent: claudebot
    -Disallow: /
    -
    -User-agent: google-extended
    -Disallow: /
    -
    -User-agent: gptbot
    -Disallow: /
    -
    -User-agent: meta-externalagent
    -Disallow: /
    -
    -User-agent: *
  • unblockjustdial.com unblocked ClaudeBot.
    Tranco rank 1,341 · November 14, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,56 +0,0 @@
    -User-agent: *
    -Disallow: /functions/
    -Disallow: /webmain/
    -Disallow: /*/wcdata/
    -Disallow: /_next/
    -Disallow: /web-worker/
    -Disallow: /us/
    -Disallow: /ca/
    -Disallow: /jd-trkr
    -Disallow: /api/
    -Disallow: /jdmart/api/
    -Disallow: /real-estate/services/api/
    -Disallow: /api/php/appUserId
    -Disallow: /gapi/ratings
    -Disallow: /api/india_api_write
    -Disallow: /api/india_api_read
    -Disallow: /bestdeals_chatbot/
    -
    -User-agent: oai-searchbot
    -Allow: /
    -
    -User-agent: chatgpt-user
    -Allow: /
    -
    -User-agent: claude-user
    -Allow: /
    -
    -User-agent: claude-searchbot
    -Allow: /
    -
    -User-agent: claudebot
    -Disallow: /
    -
    -User-agent: perplexitybot
    -Allow: /
    -
    -User-agent: perplexity-user
    -Allow: /
    -
    …
  • unblockpbs.org unblocked GPTBot, PerplexityBot, CCBot, Bytespider.
    Tranco rank 970 · November 14, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,81 +0,0 @@
    -User-agent: *
    -Disallow: /wgbh/pages/frontline/digitalnation/%3F
    -Disallow: /wgbh/pages/frontline/digitalnation/%3F*
    -Disallow: /wgbh/frontline/*elqat*
    -Disallow: /wgbh/pages/frontline/*elqat*
    -Disallow: /wgbh/pages/frontline/bushswar*
    -Disallow: /independentlens/getinvolved/cinema/
    -Disallow: /cgi-bin/*
    -Disallow: /cgi-registry/*
    -Disallow: /survey/
    -Disallow: /whatson/stations/
    -Disallow: /whatson/schedulex/
    -Disallow: /whatson/pbsyou/
    -Disallow: /whatson/dbs/
    -Disallow: /remotecontrol/bestpractices/prototype/
    -Disallow: /remotecontrol/bestpractices/jl-prototype/
    -Disallow: /pressroom/
    -Disallow: /als/shakespeare_pilot/
    -Disallow: /newshour/bb/military/military.html/
    -Disallow: /producers/webmanual/
    -Disallow: /devcon2007/
    -Disallow: /smcot/
    -Disallow: /pbsholidaycard/
    -Disallow: /gpra/
    -Disallow: /tlqa/
    -Disallow: /remote.control/
    -Disallow: /teachers/connect/
    -Disallow: /engage/sites/all/files/blog_images/monkeys.gif
    -Disallow: /teachers/edcar/
    -Disallow: /teachers/dll_research/
    -Disallow: /peerconnection/feeds/
    -Disallow: /teachers/innovators/toolkit/
    -Disallow: /mt4/
    -Disallow: /klru/mt4/
    -Disallow: /newshour/mt4/
    -Disallow: /mediashift/mt4/
    -Disallow: /kcet/mt4/
    -Disallow: /newshour/mt3/
    -Disallow: /now/mt3/
    …
  • llms.txtpewresearch.org removed its llms.txt.
    Tranco rank 1,846 · November 13, 2026
  • unblockhola.com unblocked CCBot.
    Tranco rank 1,311 · November 13, 2026
  • blockhbr.org blocked ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider.
    Tranco rank 912 · November 13, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,2 +1,3 @@
     User-agent: *
    +Allow: /sponsored/
     Disallow: /resources/
    @@ -37 +38,16 @@
     Disallow: /
    +
    +User-agent: claudebot
    +Disallow: /
    +
    +User-agent: applebot-extended
    +Disallow: /
    +
    +User-agent: bytespider
    +Disallow: /
    +
    +User-agent: meta-externalagent
    +Disallow: /
    +
    +User-agent: meta-externalfetcher
    +Disallow: /
  • unblocktechcrunch.com unblocked ChatGPT-User, Google-Extended.
    Tranco rank 633 · November 13, 2026
    robots.txt diff (AI-relevant groups)
    @@ -17,10 +17,3 @@
     
    -User-agent: chatgpt-user
    -Disallow: /
    -
    -User-agent: claude-web
     User-agent: claudebot
    -Disallow: /
    -
    -User-agent: google-extended
     Disallow: /
  • blockjustdial.com blocked ClaudeBot.
    Tranco rank 1,341 · November 12, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,56 @@
    +User-agent: *
    +Disallow: /functions/
    +Disallow: /webmain/
    +Disallow: /*/wcdata/
    +Disallow: /_next/
    +Disallow: /web-worker/
    +Disallow: /us/
    +Disallow: /ca/
    +Disallow: /jd-trkr
    +Disallow: /api/
    +Disallow: /jdmart/api/
    +Disallow: /real-estate/services/api/
    +Disallow: /api/php/appUserId
    +Disallow: /gapi/ratings
    +Disallow: /api/india_api_write
    +Disallow: /api/india_api_read
    +Disallow: /bestdeals_chatbot/
    +
    +User-agent: oai-searchbot
    +Allow: /
    +
    +User-agent: chatgpt-user
    +Allow: /
    +
    +User-agent: claude-user
    +Allow: /
    +
    +User-agent: claude-searchbot
    +Allow: /
    +
    +User-agent: claudebot
    +Disallow: /
    +
    +User-agent: perplexitybot
    +Allow: /
    +
    +User-agent: perplexity-user
    +Allow: /
    +
    …
  • blockladies.de blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
    Tranco rank 1,562 · November 12, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,28 +1,2 @@
     User-agent: *
    -Allow: /
    -Disallow: /*?*umkreis=*adresse=*
    -Disallow: /*?*suchtext=*adresse=*
    -Disallow: /*?*umkreis=70*
    -Disallow: /*?*umkreis=100*
    -Disallow: /*?*umkreis=150*
    -Disallow: /*?*umkreis=200*
    -Disallow: /*?*umkreis=300*
    -Disallow: /*?*umkreis=70$
    -Disallow: /*?*umkreis=100$
    -Disallow: /*?*umkreis=150$
    -Disallow: /*?*umkreis=200$
    -Disallow: /*?*umkreis=300$
    -Disallow: /*?page=1$
    -Disallow: /*?*page=1$
    -Disallow: /*?*page=1&*
    -Disallow: /*?*&name=*
    -Disallow: /sex-anzeigen*umkreis=*
    -Disallow: /cdn-cgi
    -Disallow: /login
    -Disallow: /api/affiliates
    -Disallow: /api/marketplace-banners
    -Disallow: /api/bookmarks
    -Disallow: /api/affiliates*
    -Disallow: /api/marketplace-banners*
    -Disallow: /api/bookmarks*
    -Disallow: /*%3D*
    +Disallow: /
  • blockgazzetta.it blocked OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, Bytespider, DuckAssistBot.
    Tranco rank 1,237 · November 12, 2026
    robots.txt diff (AI-relevant groups)
    @@ -27 +27,28 @@
     Disallow: /
    +
    +User-agent: bytespider
    +Disallow: /
    +
    +User-agent: chatgpt-user
    +Disallow: /
    +
    +User-agent: oai-searchbot
    +Disallow: /
    +
    +User-agent: claude-user
    +Disallow: /
    +
    +User-agent: claude-searchbot
    +Disallow: /
    +
    +User-agent: perplexitybot
    +Disallow: /
    +
    +User-agent: perplexity-user
    +Disallow: /
    +
    +User-agent: duckassistbot
    +Disallow: /
    +
    +User-agent: mistralai-user
    +Disallow: /
  • partialfiverr.com partially restricted OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, MistralAI-User, DuckAssistBot.
    Tranco rank 828 · November 11, 2026
    robots.txt diff (AI-relevant groups)
    @@ -56 +56,26 @@
     Disallow: /logo-maker/woo
    +
    +User-agent: google-extended
    +User-agent: oai-searchbot
    +User-agent: claude-searchbot
    +User-agent: perplexitybot
    +User-agent: mistralai-index
    +User-agent: applebot-extended
    +User-agent: youbot
    +Allow: /
    +Disallow: /orders/timeline/*
    +Disallow: */pinned_flashes/*
    +Disallow: /gigs/*/share/
    +Disallow: /gigs/*/share?*
    +Disallow: /specials/*
    +Disallow: /packages/*
    +Disallow: /categories/silly
    +Disallow: /categories/fifa
    +Disallow: /categories/Halloween
    +Disallow: /categories/Postcards
    +Disallow: /purchases
    +Disallow: /user_sessions
    +Disallow: /users/
    +Disallow: /counter/*?
    +Disallow: /collaborate/*
    +…
  • unblockcorriere.it unblocked Amazonbot.
    Tranco rank 959 · November 10, 2026
    robots.txt diff (AI-relevant groups)
    @@ -20,3 +20,3 @@
     User-agent: amazonbot
    -Disallow: /
    +Allow: /
     
  • partialllms.txtiporntv.net partially restricted GPTBot; published an llms.txt.
    Tranco rank 1,527 · November 10, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,15 @@
    +User-agent: *
    +Disallow: /player/float.php
    +Disallow: /player/html.php
    +Disallow: /*?action=*$
    +Disallow: /*mode=async*$
    +Disallow: /*function=get_block*$
    +Disallow: /404.php
    +Disallow: /captcha/*
    +Disallow: /link/*
    +Disallow: /related_videos_html/*
    +Disallow: /feedback/
    +
    +User-agent: gptbot
    +Disallow: /s/
    +Disallow: /get_file/
  • blockpartialcnbc.com blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; partially restricted Googlebot.
    Tranco rank 486 · November 10, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,77 @@
    +User-agent: googlebot
    +Disallow: /*native-android-mobile
    +Disallow: /*native-android-tablet
    +Disallow: /*mobile-native
    +Disallow: /preview/
    +Disallow: /undefined/
    +Disallow: /proplayer
    +Disallow: /appchart/*
    +Disallow: /search/*
    +Disallow: /zephr/*
    +Disallow: /blaize/*
    +
    +User-agent: gptbot
    +User-agent: perplexitybot
    +User-agent: amazonbot
    +User-agent: anthropic-ai
    +User-agent: awariorssbot
    +User-agent: awariosmartbot
    +User-agent: bytespider
    +User-agent: ccbot
    +User-agent: huggingfacebot
    +User-agent: chatgpt-user
    +User-agent: claudebot
    +User-agent: claude-web
    +User-agent: dataforseobot
    +User-agent: diffbot
    +User-agent: facebookbot
    +User-agent: google-extended
    +User-agent: google-notebooklm
    +User-agent: google-agent
    +User-agent: magpie-crawler
    +User-agent: newsnow
    +User-agent: news-please
    +User-agent: omgili
    +User-agent: omgilibot
    +User-agent: peer39_crawler
    +User-agent: peer39_crawler/1.0
    +User-agent: scrapy
    +User-agent: turnitinbot
    …
  • unblockladies.de unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
    Tranco rank 1,562 · November 9, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,2 +1,28 @@
     User-agent: *
    -Disallow: /
    +Allow: /
    +Disallow: /*?*umkreis=*adresse=*
    +Disallow: /*?*suchtext=*adresse=*
    +Disallow: /*?*umkreis=70*
    +Disallow: /*?*umkreis=100*
    +Disallow: /*?*umkreis=150*
    +Disallow: /*?*umkreis=200*
    +Disallow: /*?*umkreis=300*
    +Disallow: /*?*umkreis=70$
    +Disallow: /*?*umkreis=100$
    +Disallow: /*?*umkreis=150$
    +Disallow: /*?*umkreis=200$
    +Disallow: /*?*umkreis=300$
    +Disallow: /*?page=1$
    +Disallow: /*?*page=1$
    +Disallow: /*?*page=1&*
    +Disallow: /*?*&name=*
    +Disallow: /sex-anzeigen*umkreis=*
    +Disallow: /cdn-cgi
    +Disallow: /login
    +Disallow: /api/affiliates
    +Disallow: /api/marketplace-banners
    +Disallow: /api/bookmarks
    +Disallow: /api/affiliates*
    +Disallow: /api/marketplace-banners*
    +Disallow: /api/bookmarks*
    +Disallow: /*%3D*
  • unblockjustdial.com unblocked ClaudeBot.
    Tranco rank 1,341 · November 9, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,56 +0,0 @@
    -User-agent: *
    -Disallow: /functions/
    -Disallow: /webmain/
    -Disallow: /*/wcdata/
    -Disallow: /_next/
    -Disallow: /web-worker/
    -Disallow: /us/
    -Disallow: /ca/
    -Disallow: /jd-trkr
    -Disallow: /api/
    -Disallow: /jdmart/api/
    -Disallow: /real-estate/services/api/
    -Disallow: /api/php/appUserId
    -Disallow: /gapi/ratings
    -Disallow: /api/india_api_write
    -Disallow: /api/india_api_read
    -Disallow: /bestdeals_chatbot/
    -
    -User-agent: oai-searchbot
    -Allow: /
    -
    -User-agent: chatgpt-user
    -Allow: /
    -
    -User-agent: claude-user
    -Allow: /
    -
    -User-agent: claude-searchbot
    -Allow: /
    -
    -User-agent: claudebot
    -Disallow: /
    -
    -User-agent: perplexitybot
    -Allow: /
    -
    -User-agent: perplexity-user
    -Allow: /
    -
    …
  • blockdw.com blocked Perplexity-User.
    Tranco rank 1,390 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -63 +63,4 @@
     Disallow: /
    +
    +User-agent: perplexity-user
    +Disallow: /
  • llms.txttarget.com published an llms.txt.
    Tranco rank 608 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -0,0 +1,81 @@
    +User-agent: *
    +Disallow: /*/Ntk
    +Disallow: /*/Ntt
    +Disallow: /*/Ntx
    +Disallow: /*%7Cd_
    +Disallow: /*/schoollist/
    +Disallow: /*BTWN
    +Disallow: /[path]/
    +Disallow: /7078046/
    +Disallow: /7079046/
    +Disallow: /AddToList
    +Disallow: /AddToRegistry
    +Disallow: /admin
    +Disallow: /advancedGiftRegistrySearchView
    +Disallow: /AjaxSearchNavigationView
    +Disallow: /Allons_voter
    +Disallow: /bp/c/
    +Disallow: /bp/guest_mfg_brand
    +Disallow: /bp/p/
    +Disallow: /CallToActionModalView
    +Disallow: /cgi-bin
    +Disallow: /cgi-local
    +Disallow: /Checkout
    +Disallow: /CheckoutEditItemsDisplayView
    +Disallow: /CheckoutOrderBillingView
    +Disallow: /CheckoutOrderShippingView
    +Disallow: /CheckoutSignInView
    +Disallow: /co-
    +Disallow: /common
    +Disallow: /coupons.
    +Disallow: /custom-reviews/
    +Disallow: /data
    +Disallow: /database/philboard.mdb
    +Disallow: /deals/
    +Disallow: /dir_on_server/
    +Disallow: /EmailCartView
    +Disallow: /EnlargedImageView
    +Disallow: /ESPDisplayOptionsViewCmd
    +Disallow: /ESPModal
    …
  • unblockpartialcnbc.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; lifted partial rules for Googlebot.
    Tranco rank 486 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,77 +0,0 @@
    -User-agent: googlebot
    -Disallow: /*native-android-mobile
    -Disallow: /*native-android-tablet
    -Disallow: /*mobile-native
    -Disallow: /preview/
    -Disallow: /undefined/
    -Disallow: /proplayer
    -Disallow: /appchart/*
    -Disallow: /search/*
    -Disallow: /zephr/*
    -Disallow: /blaize/*
    -
    -User-agent: gptbot
    -User-agent: perplexitybot
    -User-agent: amazonbot
    -User-agent: anthropic-ai
    -User-agent: awariorssbot
    -User-agent: awariosmartbot
    -User-agent: bytespider
    -User-agent: ccbot
    -User-agent: huggingfacebot
    -User-agent: chatgpt-user
    -User-agent: claudebot
    -User-agent: claude-web
    -User-agent: dataforseobot
    -User-agent: diffbot
    -User-agent: facebookbot
    -User-agent: google-extended
    -User-agent: google-notebooklm
    -User-agent: google-agent
    -User-agent: magpie-crawler
    -User-agent: newsnow
    -User-agent: news-please
    -User-agent: omgili
    -User-agent: omgilibot
    -User-agent: peer39_crawler
    -User-agent: peer39_crawler/1.0
    -User-agent: scrapy
    -User-agent: turnitinbot
    …
  • unblockpartialmeraki.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot; lifted partial rules for Googlebot, Bingbot, Applebot, CCBot.
    Tranco rank 176 · November 8, 2026
    robots.txt diff (AI-relevant groups)
    @@ -1,75 +0,0 @@
    -User-agent: 360spider
    -User-agent: 360spider-image
    -User-agent: 360spider-video
    -User-agent: adsbot-google
    -User-agent: adsbot-google-mobile
    -User-agent: adsbot-google-mobile-apps
    -User-agent: adidxbot
    -User-agent: applebot
    -User-agent: applenewsbot
    -User-agent: baiduspider
    -User-agent: baiduspider-image
    -User-agent: baiduspider-news
    -User-agent: baiduspider-video
    -User-agent: bingbot
    -User-agent: bingpreview
    -User-agent: bublupbot
    -User-agent: ccbot
    -User-agent: cliqzbot
    -User-agent: coccoc
    -User-agent: coccocbot-image
    -User-agent: coccocbot-web
    -User-agent: daumoa
    -User-agent: dazoobot
    -User-agent: deusu
    -User-agent: duckduckbot
    -User-agent: duckduckgo-favicons-bot
    -User-agent: euripbot
    -User-agent: exploratodo
    -User-agent: facebot
    -User-agent: feedly
    -User-agent: findxbot
    -User-agent: googlebot
    -User-agent: googlebot-image
    -User-agent: googlebot-mobile
    -User-agent: googlebot-news
    -User-agent: googlebot-video
    -User-agent: haosouspider
    -User-agent: ichiro
    -User-agent: istellabot
    …

Weekly digests

How it works

Twice a day, at random times, we re-read the robots.txt and /llms.txt of the 2,000 most popular domains of the Tranco list (1,493 currently answer with a usable robots.txt), with the User-Agent iabig-policywatch/1.0 (+https://iabig.com/ai-policy-changes/). Each file is parsed with the RFC 9309 parser of our robots.txt tester for the 20 crawlers we track, using the same definitions as the AI Crawler Census: blocked means both the home page and an arbitrary page are disallowed, by name or through User-agent: *; partial means some paths are disallowed.

A change is published only after it has been seen in two consecutive checks, so that flapping CDNs, bot-challenge pages and temporary errors do not create false alerts; server errors, HTML pages served instead of robots.txt and unreachable hosts never change the recorded state. We keep only a normalised excerpt of the robots.txt groups that name a tracked crawler or *, and a compact diff — never the full file. Changes marked “seen between two weekly census runs” were detected by comparing the last two weekly censuses and confirmed by our first check.

Check any site's current policy with the robots.txt tester or the AI visibility checker; the overall picture for the top 10,000 is in the AI Crawler Census and the AI-Readiness Index.