Crawler by crawler
| Crawler | Newly blocked | Unblocked | Partially restricted | Restrictions lifted |
|---|---|---|---|---|
| ClaudeBot Anthropic | 4 | 4 | ||
| Bytespider ByteDance | 4 | 3 | ||
| GPTBot OpenAI | 2 | 4 | 1 | |
| Applebot-Extended Apple | 3 | 2 | 1 | |
| CCBot Common Crawl | 2 | 4 | ||
| ChatGPT-User OpenAI | 3 | 2 | 1 | |
| Google-Extended Google | 2 | 3 | 1 | |
| Perplexity-User Perplexity | 3 | 2 | 1 | |
| PerplexityBot Perplexity | 3 | 2 | 1 | |
| Amazonbot Amazon | 2 | 3 | ||
| Claude-SearchBot Anthropic | 3 | 1 | 1 | |
| Claude-User Anthropic | 3 | 1 | 1 | |
| OAI-SearchBot OpenAI | 3 | 1 | 1 | |
| meta-externalagent Meta | 3 | 2 | ||
| DuckAssistBot DuckDuckGo | 2 | 1 | 1 | |
| MistralAI-User Mistral AI | 2 | 1 | 1 | |
| Googlebot Google | 1 | 1 | 1 | |
| meta-externalfetcher Meta | 2 | 1 | ||
| Applebot Apple | 1 | 1 | ||
| Bingbot Microsoft | 1 | 1 |
All changes this week
- unblockjustdial.com unblocked ClaudeBot.
Tranco rank 1,341 · November 9, 2026robots.txt diff (AI-relevant groups)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- unblockladies.de unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Tranco rank 1,562 · November 9, 2026robots.txt diff (AI-relevant groups)
@@ -1,2 +1,28 @@ User-agent: * -Disallow: / +Allow: / +Disallow: /*?*umkreis=*adresse=* +Disallow: /*?*suchtext=*adresse=* +Disallow: /*?*umkreis=70* +Disallow: /*?*umkreis=100* +Disallow: /*?*umkreis=150* +Disallow: /*?*umkreis=200* +Disallow: /*?*umkreis=300* +Disallow: /*?*umkreis=70$ +Disallow: /*?*umkreis=100$ +Disallow: /*?*umkreis=150$ +Disallow: /*?*umkreis=200$ +Disallow: /*?*umkreis=300$ +Disallow: /*?page=1$ +Disallow: /*?*page=1$ +Disallow: /*?*page=1&* +Disallow: /*?*&name=* +Disallow: /sex-anzeigen*umkreis=* +Disallow: /cdn-cgi +Disallow: /login +Disallow: /api/affiliates +Disallow: /api/marketplace-banners +Disallow: /api/bookmarks +Disallow: /api/affiliates* +Disallow: /api/marketplace-banners* +Disallow: /api/bookmarks* +Disallow: /*%3D*
- blockpartialcnbc.com blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; partially restricted Googlebot.
Tranco rank 486 · November 10, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,77 @@ +User-agent: googlebot +Disallow: /*native-android-mobile +Disallow: /*native-android-tablet +Disallow: /*mobile-native +Disallow: /preview/ +Disallow: /undefined/ +Disallow: /proplayer +Disallow: /appchart/* +Disallow: /search/* +Disallow: /zephr/* +Disallow: /blaize/* + +User-agent: gptbot +User-agent: perplexitybot +User-agent: amazonbot +User-agent: anthropic-ai +User-agent: awariorssbot +User-agent: awariosmartbot +User-agent: bytespider +User-agent: ccbot +User-agent: huggingfacebot +User-agent: chatgpt-user +User-agent: claudebot +User-agent: claude-web +User-agent: dataforseobot +User-agent: diffbot +User-agent: facebookbot +User-agent: google-extended +User-agent: google-notebooklm +User-agent: google-agent +User-agent: magpie-crawler +User-agent: newsnow +User-agent: news-please +User-agent: omgili +User-agent: omgilibot +User-agent: peer39_crawler +User-agent: peer39_crawler/1.0 +User-agent: scrapy +User-agent: turnitinbot …
- unblockcorriere.it unblocked Amazonbot.
Tranco rank 959 · November 10, 2026robots.txt diff (AI-relevant groups)
@@ -20,3 +20,3 @@ User-agent: amazonbot -Disallow: / +Allow: /
- partialllms.txtiporntv.net partially restricted GPTBot; published an llms.txt.
Tranco rank 1,527 · November 10, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,15 @@ +User-agent: * +Disallow: /player/float.php +Disallow: /player/html.php +Disallow: /*?action=*$ +Disallow: /*mode=async*$ +Disallow: /*function=get_block*$ +Disallow: /404.php +Disallow: /captcha/* +Disallow: /link/* +Disallow: /related_videos_html/* +Disallow: /feedback/ + +User-agent: gptbot +Disallow: /s/ +Disallow: /get_file/
- partialfiverr.com partially restricted OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, MistralAI-User, DuckAssistBot.
Tranco rank 828 · November 11, 2026robots.txt diff (AI-relevant groups)
@@ -56 +56,26 @@ Disallow: /logo-maker/woo + +User-agent: google-extended +User-agent: oai-searchbot +User-agent: claude-searchbot +User-agent: perplexitybot +User-agent: mistralai-index +User-agent: applebot-extended +User-agent: youbot +Allow: / +Disallow: /orders/timeline/* +Disallow: */pinned_flashes/* +Disallow: /gigs/*/share/ +Disallow: /gigs/*/share?* +Disallow: /specials/* +Disallow: /packages/* +Disallow: /categories/silly +Disallow: /categories/fifa +Disallow: /categories/Halloween +Disallow: /categories/Postcards +Disallow: /purchases +Disallow: /user_sessions +Disallow: /users/ +Disallow: /counter/*? +Disallow: /collaborate/* +…
- blockgazzetta.it blocked OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, Bytespider, DuckAssistBot.
Tranco rank 1,237 · November 12, 2026robots.txt diff (AI-relevant groups)
@@ -27 +27,28 @@ Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: chatgpt-user +Disallow: / + +User-agent: oai-searchbot +Disallow: / + +User-agent: claude-user +Disallow: / + +User-agent: claude-searchbot +Disallow: / + +User-agent: perplexitybot +Disallow: / + +User-agent: perplexity-user +Disallow: / + +User-agent: duckassistbot +Disallow: / + +User-agent: mistralai-user +Disallow: /
- blockjustdial.com blocked ClaudeBot.
Tranco rank 1,341 · November 12, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,56 @@ +User-agent: * +Disallow: /functions/ +Disallow: /webmain/ +Disallow: /*/wcdata/ +Disallow: /_next/ +Disallow: /web-worker/ +Disallow: /us/ +Disallow: /ca/ +Disallow: /jd-trkr +Disallow: /api/ +Disallow: /jdmart/api/ +Disallow: /real-estate/services/api/ +Disallow: /api/php/appUserId +Disallow: /gapi/ratings +Disallow: /api/india_api_write +Disallow: /api/india_api_read +Disallow: /bestdeals_chatbot/ + +User-agent: oai-searchbot +Allow: / + +User-agent: chatgpt-user +Allow: / + +User-agent: claude-user +Allow: / + +User-agent: claude-searchbot +Allow: / + +User-agent: claudebot +Disallow: / + +User-agent: perplexitybot +Allow: / + +User-agent: perplexity-user +Allow: / + …
- blockladies.de blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Tranco rank 1,562 · November 12, 2026robots.txt diff (AI-relevant groups)
@@ -1,28 +1,2 @@ User-agent: * -Allow: / -Disallow: /*?*umkreis=*adresse=* -Disallow: /*?*suchtext=*adresse=* -Disallow: /*?*umkreis=70* -Disallow: /*?*umkreis=100* -Disallow: /*?*umkreis=150* -Disallow: /*?*umkreis=200* -Disallow: /*?*umkreis=300* -Disallow: /*?*umkreis=70$ -Disallow: /*?*umkreis=100$ -Disallow: /*?*umkreis=150$ -Disallow: /*?*umkreis=200$ -Disallow: /*?*umkreis=300$ -Disallow: /*?page=1$ -Disallow: /*?*page=1$ -Disallow: /*?*page=1&* -Disallow: /*?*&name=* -Disallow: /sex-anzeigen*umkreis=* -Disallow: /cdn-cgi -Disallow: /login -Disallow: /api/affiliates -Disallow: /api/marketplace-banners -Disallow: /api/bookmarks -Disallow: /api/affiliates* -Disallow: /api/marketplace-banners* -Disallow: /api/bookmarks* -Disallow: /*%3D* +Disallow: /
- unblocktechcrunch.com unblocked ChatGPT-User, Google-Extended.
Tranco rank 633 · November 13, 2026robots.txt diff (AI-relevant groups)
@@ -17,10 +17,3 @@ -User-agent: chatgpt-user -Disallow: / - -User-agent: claude-web User-agent: claudebot -Disallow: / - -User-agent: google-extended Disallow: /
- blockhbr.org blocked ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider.
Tranco rank 912 · November 13, 2026robots.txt diff (AI-relevant groups)
@@ -1,2 +1,3 @@ User-agent: * +Allow: /sponsored/ Disallow: /resources/ @@ -37 +38,16 @@ Disallow: / + +User-agent: claudebot +Disallow: / + +User-agent: applebot-extended +Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +User-agent: meta-externalfetcher +Disallow: /
- unblockhola.com unblocked CCBot.
Tranco rank 1,311 · November 13, 2026 - llms.txtpewresearch.org removed its llms.txt.
Tranco rank 1,846 · November 13, 2026 - unblockpbs.org unblocked GPTBot, PerplexityBot, CCBot, Bytespider.
Tranco rank 970 · November 14, 2026robots.txt diff (AI-relevant groups)
@@ -1,81 +0,0 @@ -User-agent: * -Disallow: /wgbh/pages/frontline/digitalnation/%3F -Disallow: /wgbh/pages/frontline/digitalnation/%3F* -Disallow: /wgbh/frontline/*elqat* -Disallow: /wgbh/pages/frontline/*elqat* -Disallow: /wgbh/pages/frontline/bushswar* -Disallow: /independentlens/getinvolved/cinema/ -Disallow: /cgi-bin/* -Disallow: /cgi-registry/* -Disallow: /survey/ -Disallow: /whatson/stations/ -Disallow: /whatson/schedulex/ -Disallow: /whatson/pbsyou/ -Disallow: /whatson/dbs/ -Disallow: /remotecontrol/bestpractices/prototype/ -Disallow: /remotecontrol/bestpractices/jl-prototype/ -Disallow: /pressroom/ -Disallow: /als/shakespeare_pilot/ -Disallow: /newshour/bb/military/military.html/ -Disallow: /producers/webmanual/ -Disallow: /devcon2007/ -Disallow: /smcot/ -Disallow: /pbsholidaycard/ -Disallow: /gpra/ -Disallow: /tlqa/ -Disallow: /remote.control/ -Disallow: /teachers/connect/ -Disallow: /engage/sites/all/files/blog_images/monkeys.gif -Disallow: /teachers/edcar/ -Disallow: /teachers/dll_research/ -Disallow: /peerconnection/feeds/ -Disallow: /teachers/innovators/toolkit/ -Disallow: /mt4/ -Disallow: /klru/mt4/ -Disallow: /newshour/mt4/ -Disallow: /mediashift/mt4/ -Disallow: /kcet/mt4/ -Disallow: /newshour/mt3/ -Disallow: /now/mt3/ …
- unblockjustdial.com unblocked ClaudeBot.
Tranco rank 1,341 · November 14, 2026robots.txt diff (AI-relevant groups)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- llms.txtunesco.org removed its llms.txt.
Tranco rank 614 · November 15, 2026 - unblockdw.com unblocked Perplexity-User.
Tranco rank 1,400 · November 15, 2026robots.txt diff (AI-relevant groups)
@@ -63,4 +63 @@ Disallow: / - -User-agent: perplexity-user -Disallow: /
- unblockloteriadehoy.com unblocked GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider.
Tranco rank 1,748 · November 15, 2026robots.txt diff (AI-relevant groups)
@@ -1,28 +1 @@ User-agent: * -Allow: / - -User-agent: amazonbot -Disallow: / - -User-agent: applebot-extended -Disallow: / - -User-agent: bytespider -Disallow: / - -User-agent: ccbot -Disallow: / - -User-agent: claudebot -Disallow: / - -User-agent: google-extended -Disallow: / - -User-agent: gptbot -Disallow: / - -User-agent: meta-externalagent -Disallow: / - -User-agent: *
- unblocksattamatkano1.me unblocked GPTBot.
Tranco rank 1,789 · November 15, 2026robots.txt diff (AI-relevant groups)
@@ -5,4 +5 @@ Allow: / - -User-agent: gptbot -Disallow: /
← Previous week · All changes · Next week →
robots.txt and llms.txt of the top 2,000 Tranco domains, checked twice a day; a change is listed once it has been seen in two consecutive checks. How it works.