Crawler by crawler, last 30 days
Number of sites whose robots.txt changed for each crawler.
| Crawler | Newly blocked | Unblocked | Partially restricted | Restrictions lifted |
|---|---|---|---|---|
| ClaudeBot Anthropic | 4 | 7 | ||
| GPTBot OpenAI | 3 | 7 | 1 | |
| Bytespider ByteDance | 4 | 6 | ||
| CCBot Common Crawl | 3 | 6 | 1 | |
| ChatGPT-User OpenAI | 4 | 5 | 1 | |
| Perplexity-User Perplexity | 4 | 5 | 1 | |
| Applebot-Extended Apple | 3 | 5 | 1 | |
| Google-Extended Google | 2 | 6 | 1 | |
| PerplexityBot Perplexity | 3 | 5 | 1 | |
| Amazonbot Amazon | 2 | 6 | ||
| Claude-SearchBot Anthropic | 3 | 4 | 1 | |
| Claude-User Anthropic | 3 | 4 | 1 | |
| OAI-SearchBot OpenAI | 3 | 4 | 1 | |
| meta-externalagent Meta | 3 | 5 | ||
| Googlebot Google | 1 | 1 | 2 | 3 |
| DuckAssistBot DuckDuckGo | 2 | 3 | 1 | |
| MistralAI-User Mistral AI | 2 | 3 | 1 | |
| meta-externalfetcher Meta | 2 | 3 | ||
| Applebot Apple | 1 | 2 | 1 | |
| Bingbot Microsoft | 1 | 1 | 1 |
Latest changes
- partialllms.txtpewresearch.org lifted partial rules for Googlebot; published an llms.txt.
Tranco rank 1,852 · November 16, 2026robots.txt diff (AI-relevant groups)
@@ -1,2 +1,5 @@ User-agent: * +Disallow: /search/ +Disallow: /search +Disallow: /?s= Disallow: /wp-admin/ @@ -4,8 +7,2 @@ Disallow: /wp-content/plugins/prc-icon-library/ - -User-agent: googlebot -Disallow: /wp-content/plugins/prc-icon-library/ -Disallow: /search/ -Disallow: /search -Disallow: /?s= Allow: /*/topline
- blockpartialarchiveofourown.org blocked GPTBot, ChatGPT-User, CCBot; partially restricted Googlebot.
Tranco rank 1,073 · November 16, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,33 @@ +User-agent: * +Disallow: /works? +Disallow: /autocomplete/ +Disallow: /downloads/ +Disallow: /external_works/ +Disallow: /bookmarks/search? +Disallow: /people/search? +Disallow: /tags/search? +Disallow: /works/search? + +User-agent: googlebot +Disallow: /autocomplete/ +Disallow: /downloads/ +Disallow: /external_works/ +Disallow: /works/*? +Disallow: /*search? +Disallow: /*?*query= +Disallow: /*?*sort_ +Disallow: /*?*selected_tags +Disallow: /*?*view_adult +Disallow: /*?*tag_id +Disallow: /*?*pseud_id +Disallow: /*?*user_id +Disallow: /*?*pseud= + +User-agent: ccbot +Disallow: / + +User-agent: gptbot +Disallow: / + +User-agent: chatgpt-user +Disallow: /
- unblockfuraffinity.net unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Tranco rank 1,228 · November 16, 2026robots.txt diff (AI-relevant groups)
@@ -1,81 +0,0 @@ -User-agent: addsearchbot -User-agent: agenttimes -User-agent: ai2bot -User-agent: ai2bot-deepresearcheval -User-agent: ai2bot-dolma -User-agent: aihitbot -User-agent: amazon-kendra -User-agent: amazonbot -User-agent: amazonbuyforme -User-agent: amzn-searchbot -User-agent: amzn-user -User-agent: andibot -User-agent: anomura -User-agent: anthropic-ai -User-agent: apifybot -User-agent: apifywebsitecontentcrawler -User-agent: applebot -User-agent: applebot-extended -User-agent: aranet-searchbot -User-agent: atlassian-bot -User-agent: awario -User-agent: azureai-searchbot -User-agent: bedrockbot -User-agent: bigsur.ai -User-agent: bravebot -User-agent: brightbot -User-agent: brightbot 1.0 -User-agent: buddybot -User-agent: bytespider -User-agent: ccbot -User-agent: channel3bot -User-agent: chatglm-spider -User-agent: chatgpt agent -User-agent: chatgpt-user -User-agent: claude-code -User-agent: claude-searchbot -User-agent: claude-user -User-agent: claude-web -User-agent: claudebot …
- unblocksattamatkano1.me unblocked GPTBot.
Tranco rank 1,789 · November 15, 2026robots.txt diff (AI-relevant groups)
@@ -5,4 +5 @@ Allow: / - -User-agent: gptbot -Disallow: /
- unblockdw.com unblocked Perplexity-User.
Tranco rank 1,400 · November 15, 2026robots.txt diff (AI-relevant groups)
@@ -63,4 +63 @@ Disallow: / - -User-agent: perplexity-user -Disallow: /
- llms.txtunesco.org removed its llms.txt.
Tranco rank 614 · November 15, 2026 - unblockloteriadehoy.com unblocked GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider.
Tranco rank 1,748 · November 15, 2026robots.txt diff (AI-relevant groups)
@@ -1,28 +1 @@ User-agent: * -Allow: / - -User-agent: amazonbot -Disallow: / - -User-agent: applebot-extended -Disallow: / - -User-agent: bytespider -Disallow: / - -User-agent: ccbot -Disallow: / - -User-agent: claudebot -Disallow: / - -User-agent: google-extended -Disallow: / - -User-agent: gptbot -Disallow: / - -User-agent: meta-externalagent -Disallow: / - -User-agent: *
- unblockjustdial.com unblocked ClaudeBot.
Tranco rank 1,341 · November 14, 2026robots.txt diff (AI-relevant groups)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- unblockpbs.org unblocked GPTBot, PerplexityBot, CCBot, Bytespider.
Tranco rank 970 · November 14, 2026robots.txt diff (AI-relevant groups)
@@ -1,81 +0,0 @@ -User-agent: * -Disallow: /wgbh/pages/frontline/digitalnation/%3F -Disallow: /wgbh/pages/frontline/digitalnation/%3F* -Disallow: /wgbh/frontline/*elqat* -Disallow: /wgbh/pages/frontline/*elqat* -Disallow: /wgbh/pages/frontline/bushswar* -Disallow: /independentlens/getinvolved/cinema/ -Disallow: /cgi-bin/* -Disallow: /cgi-registry/* -Disallow: /survey/ -Disallow: /whatson/stations/ -Disallow: /whatson/schedulex/ -Disallow: /whatson/pbsyou/ -Disallow: /whatson/dbs/ -Disallow: /remotecontrol/bestpractices/prototype/ -Disallow: /remotecontrol/bestpractices/jl-prototype/ -Disallow: /pressroom/ -Disallow: /als/shakespeare_pilot/ -Disallow: /newshour/bb/military/military.html/ -Disallow: /producers/webmanual/ -Disallow: /devcon2007/ -Disallow: /smcot/ -Disallow: /pbsholidaycard/ -Disallow: /gpra/ -Disallow: /tlqa/ -Disallow: /remote.control/ -Disallow: /teachers/connect/ -Disallow: /engage/sites/all/files/blog_images/monkeys.gif -Disallow: /teachers/edcar/ -Disallow: /teachers/dll_research/ -Disallow: /peerconnection/feeds/ -Disallow: /teachers/innovators/toolkit/ -Disallow: /mt4/ -Disallow: /klru/mt4/ -Disallow: /newshour/mt4/ -Disallow: /mediashift/mt4/ -Disallow: /kcet/mt4/ -Disallow: /newshour/mt3/ -Disallow: /now/mt3/ …
- llms.txtpewresearch.org removed its llms.txt.
Tranco rank 1,846 · November 13, 2026 - unblockhola.com unblocked CCBot.
Tranco rank 1,311 · November 13, 2026 - blockhbr.org blocked ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider.
Tranco rank 912 · November 13, 2026robots.txt diff (AI-relevant groups)
@@ -1,2 +1,3 @@ User-agent: * +Allow: /sponsored/ Disallow: /resources/ @@ -37 +38,16 @@ Disallow: / + +User-agent: claudebot +Disallow: / + +User-agent: applebot-extended +Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +User-agent: meta-externalfetcher +Disallow: /
- unblocktechcrunch.com unblocked ChatGPT-User, Google-Extended.
Tranco rank 633 · November 13, 2026robots.txt diff (AI-relevant groups)
@@ -17,10 +17,3 @@ -User-agent: chatgpt-user -Disallow: / - -User-agent: claude-web User-agent: claudebot -Disallow: / - -User-agent: google-extended Disallow: /
- blockjustdial.com blocked ClaudeBot.
Tranco rank 1,341 · November 12, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,56 @@ +User-agent: * +Disallow: /functions/ +Disallow: /webmain/ +Disallow: /*/wcdata/ +Disallow: /_next/ +Disallow: /web-worker/ +Disallow: /us/ +Disallow: /ca/ +Disallow: /jd-trkr +Disallow: /api/ +Disallow: /jdmart/api/ +Disallow: /real-estate/services/api/ +Disallow: /api/php/appUserId +Disallow: /gapi/ratings +Disallow: /api/india_api_write +Disallow: /api/india_api_read +Disallow: /bestdeals_chatbot/ + +User-agent: oai-searchbot +Allow: / + +User-agent: chatgpt-user +Allow: / + +User-agent: claude-user +Allow: / + +User-agent: claude-searchbot +Allow: / + +User-agent: claudebot +Disallow: / + +User-agent: perplexitybot +Allow: / + +User-agent: perplexity-user +Allow: / + …
- blockladies.de blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Tranco rank 1,562 · November 12, 2026robots.txt diff (AI-relevant groups)
@@ -1,28 +1,2 @@ User-agent: * -Allow: / -Disallow: /*?*umkreis=*adresse=* -Disallow: /*?*suchtext=*adresse=* -Disallow: /*?*umkreis=70* -Disallow: /*?*umkreis=100* -Disallow: /*?*umkreis=150* -Disallow: /*?*umkreis=200* -Disallow: /*?*umkreis=300* -Disallow: /*?*umkreis=70$ -Disallow: /*?*umkreis=100$ -Disallow: /*?*umkreis=150$ -Disallow: /*?*umkreis=200$ -Disallow: /*?*umkreis=300$ -Disallow: /*?page=1$ -Disallow: /*?*page=1$ -Disallow: /*?*page=1&* -Disallow: /*?*&name=* -Disallow: /sex-anzeigen*umkreis=* -Disallow: /cdn-cgi -Disallow: /login -Disallow: /api/affiliates -Disallow: /api/marketplace-banners -Disallow: /api/bookmarks -Disallow: /api/affiliates* -Disallow: /api/marketplace-banners* -Disallow: /api/bookmarks* -Disallow: /*%3D* +Disallow: /
- blockgazzetta.it blocked OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, Bytespider, DuckAssistBot.
Tranco rank 1,237 · November 12, 2026robots.txt diff (AI-relevant groups)
@@ -27 +27,28 @@ Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: chatgpt-user +Disallow: / + +User-agent: oai-searchbot +Disallow: / + +User-agent: claude-user +Disallow: / + +User-agent: claude-searchbot +Disallow: / + +User-agent: perplexitybot +Disallow: / + +User-agent: perplexity-user +Disallow: / + +User-agent: duckassistbot +Disallow: / + +User-agent: mistralai-user +Disallow: /
- partialfiverr.com partially restricted OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, MistralAI-User, DuckAssistBot.
Tranco rank 828 · November 11, 2026robots.txt diff (AI-relevant groups)
@@ -56 +56,26 @@ Disallow: /logo-maker/woo + +User-agent: google-extended +User-agent: oai-searchbot +User-agent: claude-searchbot +User-agent: perplexitybot +User-agent: mistralai-index +User-agent: applebot-extended +User-agent: youbot +Allow: / +Disallow: /orders/timeline/* +Disallow: */pinned_flashes/* +Disallow: /gigs/*/share/ +Disallow: /gigs/*/share?* +Disallow: /specials/* +Disallow: /packages/* +Disallow: /categories/silly +Disallow: /categories/fifa +Disallow: /categories/Halloween +Disallow: /categories/Postcards +Disallow: /purchases +Disallow: /user_sessions +Disallow: /users/ +Disallow: /counter/*? +Disallow: /collaborate/* +…
- unblockcorriere.it unblocked Amazonbot.
Tranco rank 959 · November 10, 2026robots.txt diff (AI-relevant groups)
@@ -20,3 +20,3 @@ User-agent: amazonbot -Disallow: / +Allow: /
- partialllms.txtiporntv.net partially restricted GPTBot; published an llms.txt.
Tranco rank 1,527 · November 10, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,15 @@ +User-agent: * +Disallow: /player/float.php +Disallow: /player/html.php +Disallow: /*?action=*$ +Disallow: /*mode=async*$ +Disallow: /*function=get_block*$ +Disallow: /404.php +Disallow: /captcha/* +Disallow: /link/* +Disallow: /related_videos_html/* +Disallow: /feedback/ + +User-agent: gptbot +Disallow: /s/ +Disallow: /get_file/
- blockpartialcnbc.com blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; partially restricted Googlebot.
Tranco rank 486 · November 10, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,77 @@ +User-agent: googlebot +Disallow: /*native-android-mobile +Disallow: /*native-android-tablet +Disallow: /*mobile-native +Disallow: /preview/ +Disallow: /undefined/ +Disallow: /proplayer +Disallow: /appchart/* +Disallow: /search/* +Disallow: /zephr/* +Disallow: /blaize/* + +User-agent: gptbot +User-agent: perplexitybot +User-agent: amazonbot +User-agent: anthropic-ai +User-agent: awariorssbot +User-agent: awariosmartbot +User-agent: bytespider +User-agent: ccbot +User-agent: huggingfacebot +User-agent: chatgpt-user +User-agent: claudebot +User-agent: claude-web +User-agent: dataforseobot +User-agent: diffbot +User-agent: facebookbot +User-agent: google-extended +User-agent: google-notebooklm +User-agent: google-agent +User-agent: magpie-crawler +User-agent: newsnow +User-agent: news-please +User-agent: omgili +User-agent: omgilibot +User-agent: peer39_crawler +User-agent: peer39_crawler/1.0 +User-agent: scrapy +User-agent: turnitinbot …
- unblockladies.de unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Tranco rank 1,562 · November 9, 2026robots.txt diff (AI-relevant groups)
@@ -1,2 +1,28 @@ User-agent: * -Disallow: / +Allow: / +Disallow: /*?*umkreis=*adresse=* +Disallow: /*?*suchtext=*adresse=* +Disallow: /*?*umkreis=70* +Disallow: /*?*umkreis=100* +Disallow: /*?*umkreis=150* +Disallow: /*?*umkreis=200* +Disallow: /*?*umkreis=300* +Disallow: /*?*umkreis=70$ +Disallow: /*?*umkreis=100$ +Disallow: /*?*umkreis=150$ +Disallow: /*?*umkreis=200$ +Disallow: /*?*umkreis=300$ +Disallow: /*?page=1$ +Disallow: /*?*page=1$ +Disallow: /*?*page=1&* +Disallow: /*?*&name=* +Disallow: /sex-anzeigen*umkreis=* +Disallow: /cdn-cgi +Disallow: /login +Disallow: /api/affiliates +Disallow: /api/marketplace-banners +Disallow: /api/bookmarks +Disallow: /api/affiliates* +Disallow: /api/marketplace-banners* +Disallow: /api/bookmarks* +Disallow: /*%3D*
- unblockjustdial.com unblocked ClaudeBot.
Tranco rank 1,341 · November 9, 2026robots.txt diff (AI-relevant groups)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- blockdw.com blocked Perplexity-User.
Tranco rank 1,390 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -63 +63,4 @@ Disallow: / + +User-agent: perplexity-user +Disallow: /
- llms.txttarget.com published an llms.txt.
Tranco rank 608 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,81 @@ +User-agent: * +Disallow: /*/Ntk +Disallow: /*/Ntt +Disallow: /*/Ntx +Disallow: /*%7Cd_ +Disallow: /*/schoollist/ +Disallow: /*BTWN +Disallow: /[path]/ +Disallow: /7078046/ +Disallow: /7079046/ +Disallow: /AddToList +Disallow: /AddToRegistry +Disallow: /admin +Disallow: /advancedGiftRegistrySearchView +Disallow: /AjaxSearchNavigationView +Disallow: /Allons_voter +Disallow: /bp/c/ +Disallow: /bp/guest_mfg_brand +Disallow: /bp/p/ +Disallow: /CallToActionModalView +Disallow: /cgi-bin +Disallow: /cgi-local +Disallow: /Checkout +Disallow: /CheckoutEditItemsDisplayView +Disallow: /CheckoutOrderBillingView +Disallow: /CheckoutOrderShippingView +Disallow: /CheckoutSignInView +Disallow: /co- +Disallow: /common +Disallow: /coupons. +Disallow: /custom-reviews/ +Disallow: /data +Disallow: /database/philboard.mdb +Disallow: /deals/ +Disallow: /dir_on_server/ +Disallow: /EmailCartView +Disallow: /EnlargedImageView +Disallow: /ESPDisplayOptionsViewCmd +Disallow: /ESPModal …
- unblockpartialcnbc.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; lifted partial rules for Googlebot.
Tranco rank 486 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -1,77 +0,0 @@ -User-agent: googlebot -Disallow: /*native-android-mobile -Disallow: /*native-android-tablet -Disallow: /*mobile-native -Disallow: /preview/ -Disallow: /undefined/ -Disallow: /proplayer -Disallow: /appchart/* -Disallow: /search/* -Disallow: /zephr/* -Disallow: /blaize/* - -User-agent: gptbot -User-agent: perplexitybot -User-agent: amazonbot -User-agent: anthropic-ai -User-agent: awariorssbot -User-agent: awariosmartbot -User-agent: bytespider -User-agent: ccbot -User-agent: huggingfacebot -User-agent: chatgpt-user -User-agent: claudebot -User-agent: claude-web -User-agent: dataforseobot -User-agent: diffbot -User-agent: facebookbot -User-agent: google-extended -User-agent: google-notebooklm -User-agent: google-agent -User-agent: magpie-crawler -User-agent: newsnow -User-agent: news-please -User-agent: omgili -User-agent: omgilibot -User-agent: peer39_crawler -User-agent: peer39_crawler/1.0 -User-agent: scrapy -User-agent: turnitinbot …
- unblockpartialmeraki.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot; lifted partial rules for Googlebot, Bingbot, Applebot, CCBot.
Tranco rank 176 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -1,75 +0,0 @@ -User-agent: 360spider -User-agent: 360spider-image -User-agent: 360spider-video -User-agent: adsbot-google -User-agent: adsbot-google-mobile -User-agent: adsbot-google-mobile-apps -User-agent: adidxbot -User-agent: applebot -User-agent: applenewsbot -User-agent: baiduspider -User-agent: baiduspider-image -User-agent: baiduspider-news -User-agent: baiduspider-video -User-agent: bingbot -User-agent: bingpreview -User-agent: bublupbot -User-agent: ccbot -User-agent: cliqzbot -User-agent: coccoc -User-agent: coccocbot-image -User-agent: coccocbot-web -User-agent: daumoa -User-agent: dazoobot -User-agent: deusu -User-agent: duckduckbot -User-agent: duckduckgo-favicons-bot -User-agent: euripbot -User-agent: exploratodo -User-agent: facebot -User-agent: feedly -User-agent: findxbot -User-agent: googlebot -User-agent: googlebot-image -User-agent: googlebot-mobile -User-agent: googlebot-news -User-agent: googlebot-video -User-agent: haosouspider -User-agent: ichiro -User-agent: istellabot …
Weekly digests
- AI policy changes, week 47 of 2026 (November 16, 2026 – November 22, 2026) — 3 change(s) on 3 top domains: 1 newly blocking AI crawlers, 1 unblocking, 1 new llms.txt.
- AI policy changes, week 46 of 2026 (November 9, 2026 – November 15, 2026) — 19 change(s) on 16 top domains: 5 newly blocking AI crawlers, 10 unblocking, 1 new llms.txt.
- AI policy changes, week 45 of 2026 (November 2, 2026 – November 8, 2026) — 4 change(s) on 4 top domains: 1 newly blocking AI crawlers, 2 unblocking, 1 new llms.txt.
How it works
Twice a day, at random times, we re-read the robots.txt and /llms.txt of the 2,000 most popular domains of the Tranco list (1,493 currently answer with a usable robots.txt), with the User-Agent iabig-policywatch/1.0 (+https://iabig.com/ai-policy-changes/). Each file is parsed with the RFC 9309 parser of our robots.txt tester for the 20 crawlers we track, using the same definitions as the AI Crawler Census: blocked means both the home page and an arbitrary page are disallowed, by name or through User-agent: *; partial means some paths are disallowed.
A change is published only after it has been seen in two consecutive checks, so that flapping CDNs, bot-challenge pages and temporary errors do not create false alerts; server errors, HTML pages served instead of robots.txt and unreachable hosts never change the recorded state. We keep only a normalised excerpt of the robots.txt groups that name a tracked crawler or *, and a compact diff — never the full file. Changes marked “seen between two weekly census runs” were detected by comparing the last two weekly censuses and confirmed by our first check.