Crawler by crawler
| Crawler | Newly blocked | Unblocked | Partially restricted | Restrictions lifted |
|---|---|---|---|---|
| Perplexity-User Perplexity | 1 | 2 | ||
| Amazonbot Amazon | 2 | |||
| Applebot-Extended Apple | 2 | |||
| Bytespider ByteDance | 2 | |||
| CCBot Common Crawl | 1 | 1 | ||
| ChatGPT-User OpenAI | 2 | |||
| Claude-SearchBot Anthropic | 2 | |||
| Claude-User Anthropic | 2 | |||
| ClaudeBot Anthropic | 2 | |||
| GPTBot OpenAI | 2 | |||
| Google-Extended Google | 2 | |||
| Googlebot Google | 2 | |||
| OAI-SearchBot OpenAI | 2 | |||
| PerplexityBot Perplexity | 2 | |||
| meta-externalagent Meta | 2 | |||
| Applebot Apple | 1 | |||
| Bingbot Microsoft | 1 | |||
| DuckAssistBot DuckDuckGo | 1 | |||
| MistralAI-User Mistral AI | 1 | |||
| meta-externalfetcher Meta | 1 |
All changes this week
- unblockpartialmeraki.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot; lifted partial rules for Googlebot, Bingbot, Applebot, CCBot.
Tranco rank 176 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -1,75 +0,0 @@ -User-agent: 360spider -User-agent: 360spider-image -User-agent: 360spider-video -User-agent: adsbot-google -User-agent: adsbot-google-mobile -User-agent: adsbot-google-mobile-apps -User-agent: adidxbot -User-agent: applebot -User-agent: applenewsbot -User-agent: baiduspider -User-agent: baiduspider-image -User-agent: baiduspider-news -User-agent: baiduspider-video -User-agent: bingbot -User-agent: bingpreview -User-agent: bublupbot -User-agent: ccbot -User-agent: cliqzbot -User-agent: coccoc -User-agent: coccocbot-image -User-agent: coccocbot-web -User-agent: daumoa -User-agent: dazoobot -User-agent: deusu -User-agent: duckduckbot -User-agent: duckduckgo-favicons-bot -User-agent: euripbot -User-agent: exploratodo -User-agent: facebot -User-agent: feedly -User-agent: findxbot -User-agent: googlebot -User-agent: googlebot-image -User-agent: googlebot-mobile -User-agent: googlebot-news -User-agent: googlebot-video -User-agent: haosouspider -User-agent: ichiro -User-agent: istellabot …
- unblockpartialcnbc.com unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; lifted partial rules for Googlebot.
Tranco rank 486 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -1,77 +0,0 @@ -User-agent: googlebot -Disallow: /*native-android-mobile -Disallow: /*native-android-tablet -Disallow: /*mobile-native -Disallow: /preview/ -Disallow: /undefined/ -Disallow: /proplayer -Disallow: /appchart/* -Disallow: /search/* -Disallow: /zephr/* -Disallow: /blaize/* - -User-agent: gptbot -User-agent: perplexitybot -User-agent: amazonbot -User-agent: anthropic-ai -User-agent: awariorssbot -User-agent: awariosmartbot -User-agent: bytespider -User-agent: ccbot -User-agent: huggingfacebot -User-agent: chatgpt-user -User-agent: claudebot -User-agent: claude-web -User-agent: dataforseobot -User-agent: diffbot -User-agent: facebookbot -User-agent: google-extended -User-agent: google-notebooklm -User-agent: google-agent -User-agent: magpie-crawler -User-agent: newsnow -User-agent: news-please -User-agent: omgili -User-agent: omgilibot -User-agent: peer39_crawler -User-agent: peer39_crawler/1.0 -User-agent: scrapy -User-agent: turnitinbot …
- llms.txttarget.com published an llms.txt.
Tranco rank 608 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -0,0 +1,81 @@ +User-agent: * +Disallow: /*/Ntk +Disallow: /*/Ntt +Disallow: /*/Ntx +Disallow: /*%7Cd_ +Disallow: /*/schoollist/ +Disallow: /*BTWN +Disallow: /[path]/ +Disallow: /7078046/ +Disallow: /7079046/ +Disallow: /AddToList +Disallow: /AddToRegistry +Disallow: /admin +Disallow: /advancedGiftRegistrySearchView +Disallow: /AjaxSearchNavigationView +Disallow: /Allons_voter +Disallow: /bp/c/ +Disallow: /bp/guest_mfg_brand +Disallow: /bp/p/ +Disallow: /CallToActionModalView +Disallow: /cgi-bin +Disallow: /cgi-local +Disallow: /Checkout +Disallow: /CheckoutEditItemsDisplayView +Disallow: /CheckoutOrderBillingView +Disallow: /CheckoutOrderShippingView +Disallow: /CheckoutSignInView +Disallow: /co- +Disallow: /common +Disallow: /coupons. +Disallow: /custom-reviews/ +Disallow: /data +Disallow: /database/philboard.mdb +Disallow: /deals/ +Disallow: /dir_on_server/ +Disallow: /EmailCartView +Disallow: /EnlargedImageView +Disallow: /ESPDisplayOptionsViewCmd +Disallow: /ESPModal …
- blockdw.com blocked Perplexity-User.
Tranco rank 1,390 · November 8, 2026robots.txt diff (AI-relevant groups)
@@ -63 +63,4 @@ Disallow: / + +User-agent: perplexity-user +Disallow: /
robots.txt and llms.txt of the top 2,000 Tranco domains, checked twice a day; a change is listed once it has been seen in two consecutive checks. How it works.