Robot par robot
| Robot | Nouveaux blocages | Déblocages | Restreint en partie | Restrictions levées |
|---|---|---|---|---|
| ClaudeBot Anthropic | 4 | 4 | ||
| Bytespider ByteDance | 4 | 3 | ||
| GPTBot OpenAI | 2 | 4 | 1 | |
| Applebot-Extended Apple | 3 | 2 | 1 | |
| CCBot Common Crawl | 2 | 4 | ||
| ChatGPT-User OpenAI | 3 | 2 | 1 | |
| Google-Extended Google | 2 | 3 | 1 | |
| Perplexity-User Perplexity | 3 | 2 | 1 | |
| PerplexityBot Perplexity | 3 | 2 | 1 | |
| Amazonbot Amazon | 2 | 3 | ||
| Claude-SearchBot Anthropic | 3 | 1 | 1 | |
| Claude-User Anthropic | 3 | 1 | 1 | |
| OAI-SearchBot OpenAI | 3 | 1 | 1 | |
| meta-externalagent Meta | 3 | 2 | ||
| DuckAssistBot DuckDuckGo | 2 | 1 | 1 | |
| MistralAI-User Mistral AI | 2 | 1 | 1 | |
| Googlebot Google | 1 | 1 | 1 | |
| meta-externalfetcher Meta | 2 | 1 | ||
| Applebot Apple | 1 | 1 | ||
| Bingbot Microsoft | 1 | 1 |
Tous les changements de la semaine
- déblocagejustdial.com débloque ClaudeBot.
Rang Tranco 1 341 · 9 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- déblocageladies.de débloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Rang Tranco 1 562 · 9 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,2 +1,28 @@ User-agent: * -Disallow: / +Allow: / +Disallow: /*?*umkreis=*adresse=* +Disallow: /*?*suchtext=*adresse=* +Disallow: /*?*umkreis=70* +Disallow: /*?*umkreis=100* +Disallow: /*?*umkreis=150* +Disallow: /*?*umkreis=200* +Disallow: /*?*umkreis=300* +Disallow: /*?*umkreis=70$ +Disallow: /*?*umkreis=100$ +Disallow: /*?*umkreis=150$ +Disallow: /*?*umkreis=200$ +Disallow: /*?*umkreis=300$ +Disallow: /*?page=1$ +Disallow: /*?*page=1$ +Disallow: /*?*page=1&* +Disallow: /*?*&name=* +Disallow: /sex-anzeigen*umkreis=* +Disallow: /cdn-cgi +Disallow: /login +Disallow: /api/affiliates +Disallow: /api/marketplace-banners +Disallow: /api/bookmarks +Disallow: /api/affiliates* +Disallow: /api/marketplace-banners* +Disallow: /api/bookmarks* +Disallow: /*%3D*
- blocagepartielcnbc.com bloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; restreint en partie Googlebot.
Rang Tranco 486 · 10 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,77 @@ +User-agent: googlebot +Disallow: /*native-android-mobile +Disallow: /*native-android-tablet +Disallow: /*mobile-native +Disallow: /preview/ +Disallow: /undefined/ +Disallow: /proplayer +Disallow: /appchart/* +Disallow: /search/* +Disallow: /zephr/* +Disallow: /blaize/* + +User-agent: gptbot +User-agent: perplexitybot +User-agent: amazonbot +User-agent: anthropic-ai +User-agent: awariorssbot +User-agent: awariosmartbot +User-agent: bytespider +User-agent: ccbot +User-agent: huggingfacebot +User-agent: chatgpt-user +User-agent: claudebot +User-agent: claude-web +User-agent: dataforseobot +User-agent: diffbot +User-agent: facebookbot +User-agent: google-extended +User-agent: google-notebooklm +User-agent: google-agent +User-agent: magpie-crawler +User-agent: newsnow +User-agent: news-please +User-agent: omgili +User-agent: omgilibot +User-agent: peer39_crawler +User-agent: peer39_crawler/1.0 +User-agent: scrapy +User-agent: turnitinbot …
- déblocagecorriere.it débloque Amazonbot.
Rang Tranco 959 · 10 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -20,3 +20,3 @@ User-agent: amazonbot -Disallow: / +Allow: /
- partielllms.txtiporntv.net restreint en partie GPTBot; publie un llms.txt.
Rang Tranco 1 527 · 10 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,15 @@ +User-agent: * +Disallow: /player/float.php +Disallow: /player/html.php +Disallow: /*?action=*$ +Disallow: /*mode=async*$ +Disallow: /*function=get_block*$ +Disallow: /404.php +Disallow: /captcha/* +Disallow: /link/* +Disallow: /related_videos_html/* +Disallow: /feedback/ + +User-agent: gptbot +Disallow: /s/ +Disallow: /get_file/
- partielfiverr.com restreint en partie OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, MistralAI-User, DuckAssistBot.
Rang Tranco 828 · 11 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -56 +56,26 @@ Disallow: /logo-maker/woo + +User-agent: google-extended +User-agent: oai-searchbot +User-agent: claude-searchbot +User-agent: perplexitybot +User-agent: mistralai-index +User-agent: applebot-extended +User-agent: youbot +Allow: / +Disallow: /orders/timeline/* +Disallow: */pinned_flashes/* +Disallow: /gigs/*/share/ +Disallow: /gigs/*/share?* +Disallow: /specials/* +Disallow: /packages/* +Disallow: /categories/silly +Disallow: /categories/fifa +Disallow: /categories/Halloween +Disallow: /categories/Postcards +Disallow: /purchases +Disallow: /user_sessions +Disallow: /users/ +Disallow: /counter/*? +Disallow: /collaborate/* +…
- blocagegazzetta.it bloque OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, Bytespider, DuckAssistBot.
Rang Tranco 1 237 · 12 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -27 +27,28 @@ Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: chatgpt-user +Disallow: / + +User-agent: oai-searchbot +Disallow: / + +User-agent: claude-user +Disallow: / + +User-agent: claude-searchbot +Disallow: / + +User-agent: perplexitybot +Disallow: / + +User-agent: perplexity-user +Disallow: / + +User-agent: duckassistbot +Disallow: / + +User-agent: mistralai-user +Disallow: /
- blocagejustdial.com bloque ClaudeBot.
Rang Tranco 1 341 · 12 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,56 @@ +User-agent: * +Disallow: /functions/ +Disallow: /webmain/ +Disallow: /*/wcdata/ +Disallow: /_next/ +Disallow: /web-worker/ +Disallow: /us/ +Disallow: /ca/ +Disallow: /jd-trkr +Disallow: /api/ +Disallow: /jdmart/api/ +Disallow: /real-estate/services/api/ +Disallow: /api/php/appUserId +Disallow: /gapi/ratings +Disallow: /api/india_api_write +Disallow: /api/india_api_read +Disallow: /bestdeals_chatbot/ + +User-agent: oai-searchbot +Allow: / + +User-agent: chatgpt-user +Allow: / + +User-agent: claude-user +Allow: / + +User-agent: claude-searchbot +Allow: / + +User-agent: claudebot +Disallow: / + +User-agent: perplexitybot +Allow: / + +User-agent: perplexity-user +Allow: / + …
- blocageladies.de bloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Rang Tranco 1 562 · 12 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,28 +1,2 @@ User-agent: * -Allow: / -Disallow: /*?*umkreis=*adresse=* -Disallow: /*?*suchtext=*adresse=* -Disallow: /*?*umkreis=70* -Disallow: /*?*umkreis=100* -Disallow: /*?*umkreis=150* -Disallow: /*?*umkreis=200* -Disallow: /*?*umkreis=300* -Disallow: /*?*umkreis=70$ -Disallow: /*?*umkreis=100$ -Disallow: /*?*umkreis=150$ -Disallow: /*?*umkreis=200$ -Disallow: /*?*umkreis=300$ -Disallow: /*?page=1$ -Disallow: /*?*page=1$ -Disallow: /*?*page=1&* -Disallow: /*?*&name=* -Disallow: /sex-anzeigen*umkreis=* -Disallow: /cdn-cgi -Disallow: /login -Disallow: /api/affiliates -Disallow: /api/marketplace-banners -Disallow: /api/bookmarks -Disallow: /api/affiliates* -Disallow: /api/marketplace-banners* -Disallow: /api/bookmarks* -Disallow: /*%3D* +Disallow: /
- déblocagetechcrunch.com débloque ChatGPT-User, Google-Extended.
Rang Tranco 633 · 13 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -17,10 +17,3 @@ -User-agent: chatgpt-user -Disallow: / - -User-agent: claude-web User-agent: claudebot -Disallow: / - -User-agent: google-extended Disallow: /
- blocagehbr.org bloque ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider.
Rang Tranco 912 · 13 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,2 +1,3 @@ User-agent: * +Allow: /sponsored/ Disallow: /resources/ @@ -37 +38,16 @@ Disallow: / + +User-agent: claudebot +Disallow: / + +User-agent: applebot-extended +Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +User-agent: meta-externalfetcher +Disallow: /
- déblocagehola.com débloque CCBot.
Rang Tranco 1 311 · 13 novembre 2026 - llms.txtpewresearch.org retire son llms.txt.
Rang Tranco 1 846 · 13 novembre 2026 - déblocagepbs.org débloque GPTBot, PerplexityBot, CCBot, Bytespider.
Rang Tranco 970 · 14 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,81 +0,0 @@ -User-agent: * -Disallow: /wgbh/pages/frontline/digitalnation/%3F -Disallow: /wgbh/pages/frontline/digitalnation/%3F* -Disallow: /wgbh/frontline/*elqat* -Disallow: /wgbh/pages/frontline/*elqat* -Disallow: /wgbh/pages/frontline/bushswar* -Disallow: /independentlens/getinvolved/cinema/ -Disallow: /cgi-bin/* -Disallow: /cgi-registry/* -Disallow: /survey/ -Disallow: /whatson/stations/ -Disallow: /whatson/schedulex/ -Disallow: /whatson/pbsyou/ -Disallow: /whatson/dbs/ -Disallow: /remotecontrol/bestpractices/prototype/ -Disallow: /remotecontrol/bestpractices/jl-prototype/ -Disallow: /pressroom/ -Disallow: /als/shakespeare_pilot/ -Disallow: /newshour/bb/military/military.html/ -Disallow: /producers/webmanual/ -Disallow: /devcon2007/ -Disallow: /smcot/ -Disallow: /pbsholidaycard/ -Disallow: /gpra/ -Disallow: /tlqa/ -Disallow: /remote.control/ -Disallow: /teachers/connect/ -Disallow: /engage/sites/all/files/blog_images/monkeys.gif -Disallow: /teachers/edcar/ -Disallow: /teachers/dll_research/ -Disallow: /peerconnection/feeds/ -Disallow: /teachers/innovators/toolkit/ -Disallow: /mt4/ -Disallow: /klru/mt4/ -Disallow: /newshour/mt4/ -Disallow: /mediashift/mt4/ -Disallow: /kcet/mt4/ -Disallow: /newshour/mt3/ -Disallow: /now/mt3/ …
- déblocagejustdial.com débloque ClaudeBot.
Rang Tranco 1 341 · 14 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- llms.txtunesco.org retire son llms.txt.
Rang Tranco 614 · 15 novembre 2026 - déblocagedw.com débloque Perplexity-User.
Rang Tranco 1 400 · 15 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -63,4 +63 @@ Disallow: / - -User-agent: perplexity-user -Disallow: /
- déblocageloteriadehoy.com débloque GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider.
Rang Tranco 1 748 · 15 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,28 +1 @@ User-agent: * -Allow: / - -User-agent: amazonbot -Disallow: / - -User-agent: applebot-extended -Disallow: / - -User-agent: bytespider -Disallow: / - -User-agent: ccbot -Disallow: / - -User-agent: claudebot -Disallow: / - -User-agent: google-extended -Disallow: / - -User-agent: gptbot -Disallow: / - -User-agent: meta-externalagent -Disallow: / - -User-agent: *
- déblocagesattamatkano1.me débloque GPTBot.
Rang Tranco 1 789 · 15 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -5,4 +5 @@ Allow: / - -User-agent: gptbot -Disallow: /
← Semaine précédente · Tous les changements · Semaine suivante →
robots.txt et llms.txt des 2 000 premiers domaines Tranco, vérifiés deux fois par jour ; un changement figure ici après avoir été observé lors de deux vérifications consécutives. Fonctionnement.