Robot par robot, 30 derniers jours
Nombre de sites dont le robots.txt a changé pour chaque robot.
| Robot | Nouveaux blocages | Déblocages | Restreint en partie | Restrictions levées |
|---|---|---|---|---|
| ClaudeBot Anthropic | 4 | 7 | ||
| GPTBot OpenAI | 3 | 7 | 1 | |
| Bytespider ByteDance | 4 | 6 | ||
| CCBot Common Crawl | 3 | 6 | 1 | |
| ChatGPT-User OpenAI | 4 | 5 | 1 | |
| Perplexity-User Perplexity | 4 | 5 | 1 | |
| Applebot-Extended Apple | 3 | 5 | 1 | |
| Google-Extended Google | 2 | 6 | 1 | |
| PerplexityBot Perplexity | 3 | 5 | 1 | |
| Amazonbot Amazon | 2 | 6 | ||
| Claude-SearchBot Anthropic | 3 | 4 | 1 | |
| Claude-User Anthropic | 3 | 4 | 1 | |
| OAI-SearchBot OpenAI | 3 | 4 | 1 | |
| meta-externalagent Meta | 3 | 5 | ||
| Googlebot Google | 1 | 1 | 2 | 3 |
| DuckAssistBot DuckDuckGo | 2 | 3 | 1 | |
| MistralAI-User Mistral AI | 2 | 3 | 1 | |
| meta-externalfetcher Meta | 2 | 3 | ||
| Applebot Apple | 1 | 2 | 1 | |
| Bingbot Microsoft | 1 | 1 | 1 |
Derniers changements
- partielllms.txtpewresearch.org lève des restrictions partielles pour Googlebot; publie un llms.txt.
Rang Tranco 1 852 · 16 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,2 +1,5 @@ User-agent: * +Disallow: /search/ +Disallow: /search +Disallow: /?s= Disallow: /wp-admin/ @@ -4,8 +7,2 @@ Disallow: /wp-content/plugins/prc-icon-library/ - -User-agent: googlebot -Disallow: /wp-content/plugins/prc-icon-library/ -Disallow: /search/ -Disallow: /search -Disallow: /?s= Allow: /*/topline
- blocagepartielarchiveofourown.org bloque GPTBot, ChatGPT-User, CCBot; restreint en partie Googlebot.
Rang Tranco 1 073 · 16 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,33 @@ +User-agent: * +Disallow: /works? +Disallow: /autocomplete/ +Disallow: /downloads/ +Disallow: /external_works/ +Disallow: /bookmarks/search? +Disallow: /people/search? +Disallow: /tags/search? +Disallow: /works/search? + +User-agent: googlebot +Disallow: /autocomplete/ +Disallow: /downloads/ +Disallow: /external_works/ +Disallow: /works/*? +Disallow: /*search? +Disallow: /*?*query= +Disallow: /*?*sort_ +Disallow: /*?*selected_tags +Disallow: /*?*view_adult +Disallow: /*?*tag_id +Disallow: /*?*pseud_id +Disallow: /*?*user_id +Disallow: /*?*pseud= + +User-agent: ccbot +Disallow: / + +User-agent: gptbot +Disallow: / + +User-agent: chatgpt-user +Disallow: /
- déblocagefuraffinity.net débloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Rang Tranco 1 228 · 16 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,81 +0,0 @@ -User-agent: addsearchbot -User-agent: agenttimes -User-agent: ai2bot -User-agent: ai2bot-deepresearcheval -User-agent: ai2bot-dolma -User-agent: aihitbot -User-agent: amazon-kendra -User-agent: amazonbot -User-agent: amazonbuyforme -User-agent: amzn-searchbot -User-agent: amzn-user -User-agent: andibot -User-agent: anomura -User-agent: anthropic-ai -User-agent: apifybot -User-agent: apifywebsitecontentcrawler -User-agent: applebot -User-agent: applebot-extended -User-agent: aranet-searchbot -User-agent: atlassian-bot -User-agent: awario -User-agent: azureai-searchbot -User-agent: bedrockbot -User-agent: bigsur.ai -User-agent: bravebot -User-agent: brightbot -User-agent: brightbot 1.0 -User-agent: buddybot -User-agent: bytespider -User-agent: ccbot -User-agent: channel3bot -User-agent: chatglm-spider -User-agent: chatgpt agent -User-agent: chatgpt-user -User-agent: claude-code -User-agent: claude-searchbot -User-agent: claude-user -User-agent: claude-web -User-agent: claudebot …
- déblocagesattamatkano1.me débloque GPTBot.
Rang Tranco 1 789 · 15 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -5,4 +5 @@ Allow: / - -User-agent: gptbot -Disallow: /
- déblocagedw.com débloque Perplexity-User.
Rang Tranco 1 400 · 15 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -63,4 +63 @@ Disallow: / - -User-agent: perplexity-user -Disallow: /
- llms.txtunesco.org retire son llms.txt.
Rang Tranco 614 · 15 novembre 2026 - déblocageloteriadehoy.com débloque GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider.
Rang Tranco 1 748 · 15 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,28 +1 @@ User-agent: * -Allow: / - -User-agent: amazonbot -Disallow: / - -User-agent: applebot-extended -Disallow: / - -User-agent: bytespider -Disallow: / - -User-agent: ccbot -Disallow: / - -User-agent: claudebot -Disallow: / - -User-agent: google-extended -Disallow: / - -User-agent: gptbot -Disallow: / - -User-agent: meta-externalagent -Disallow: / - -User-agent: *
- déblocagejustdial.com débloque ClaudeBot.
Rang Tranco 1 341 · 14 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- déblocagepbs.org débloque GPTBot, PerplexityBot, CCBot, Bytespider.
Rang Tranco 970 · 14 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,81 +0,0 @@ -User-agent: * -Disallow: /wgbh/pages/frontline/digitalnation/%3F -Disallow: /wgbh/pages/frontline/digitalnation/%3F* -Disallow: /wgbh/frontline/*elqat* -Disallow: /wgbh/pages/frontline/*elqat* -Disallow: /wgbh/pages/frontline/bushswar* -Disallow: /independentlens/getinvolved/cinema/ -Disallow: /cgi-bin/* -Disallow: /cgi-registry/* -Disallow: /survey/ -Disallow: /whatson/stations/ -Disallow: /whatson/schedulex/ -Disallow: /whatson/pbsyou/ -Disallow: /whatson/dbs/ -Disallow: /remotecontrol/bestpractices/prototype/ -Disallow: /remotecontrol/bestpractices/jl-prototype/ -Disallow: /pressroom/ -Disallow: /als/shakespeare_pilot/ -Disallow: /newshour/bb/military/military.html/ -Disallow: /producers/webmanual/ -Disallow: /devcon2007/ -Disallow: /smcot/ -Disallow: /pbsholidaycard/ -Disallow: /gpra/ -Disallow: /tlqa/ -Disallow: /remote.control/ -Disallow: /teachers/connect/ -Disallow: /engage/sites/all/files/blog_images/monkeys.gif -Disallow: /teachers/edcar/ -Disallow: /teachers/dll_research/ -Disallow: /peerconnection/feeds/ -Disallow: /teachers/innovators/toolkit/ -Disallow: /mt4/ -Disallow: /klru/mt4/ -Disallow: /newshour/mt4/ -Disallow: /mediashift/mt4/ -Disallow: /kcet/mt4/ -Disallow: /newshour/mt3/ -Disallow: /now/mt3/ …
- llms.txtpewresearch.org retire son llms.txt.
Rang Tranco 1 846 · 13 novembre 2026 - déblocagehola.com débloque CCBot.
Rang Tranco 1 311 · 13 novembre 2026 - blocagehbr.org bloque ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider.
Rang Tranco 912 · 13 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,2 +1,3 @@ User-agent: * +Allow: /sponsored/ Disallow: /resources/ @@ -37 +38,16 @@ Disallow: / + +User-agent: claudebot +Disallow: / + +User-agent: applebot-extended +Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +User-agent: meta-externalfetcher +Disallow: /
- déblocagetechcrunch.com débloque ChatGPT-User, Google-Extended.
Rang Tranco 633 · 13 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -17,10 +17,3 @@ -User-agent: chatgpt-user -Disallow: / - -User-agent: claude-web User-agent: claudebot -Disallow: / - -User-agent: google-extended Disallow: /
- blocagejustdial.com bloque ClaudeBot.
Rang Tranco 1 341 · 12 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,56 @@ +User-agent: * +Disallow: /functions/ +Disallow: /webmain/ +Disallow: /*/wcdata/ +Disallow: /_next/ +Disallow: /web-worker/ +Disallow: /us/ +Disallow: /ca/ +Disallow: /jd-trkr +Disallow: /api/ +Disallow: /jdmart/api/ +Disallow: /real-estate/services/api/ +Disallow: /api/php/appUserId +Disallow: /gapi/ratings +Disallow: /api/india_api_write +Disallow: /api/india_api_read +Disallow: /bestdeals_chatbot/ + +User-agent: oai-searchbot +Allow: / + +User-agent: chatgpt-user +Allow: / + +User-agent: claude-user +Allow: / + +User-agent: claude-searchbot +Allow: / + +User-agent: claudebot +Disallow: / + +User-agent: perplexitybot +Allow: / + +User-agent: perplexity-user +Allow: / + …
- blocageladies.de bloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Rang Tranco 1 562 · 12 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,28 +1,2 @@ User-agent: * -Allow: / -Disallow: /*?*umkreis=*adresse=* -Disallow: /*?*suchtext=*adresse=* -Disallow: /*?*umkreis=70* -Disallow: /*?*umkreis=100* -Disallow: /*?*umkreis=150* -Disallow: /*?*umkreis=200* -Disallow: /*?*umkreis=300* -Disallow: /*?*umkreis=70$ -Disallow: /*?*umkreis=100$ -Disallow: /*?*umkreis=150$ -Disallow: /*?*umkreis=200$ -Disallow: /*?*umkreis=300$ -Disallow: /*?page=1$ -Disallow: /*?*page=1$ -Disallow: /*?*page=1&* -Disallow: /*?*&name=* -Disallow: /sex-anzeigen*umkreis=* -Disallow: /cdn-cgi -Disallow: /login -Disallow: /api/affiliates -Disallow: /api/marketplace-banners -Disallow: /api/bookmarks -Disallow: /api/affiliates* -Disallow: /api/marketplace-banners* -Disallow: /api/bookmarks* -Disallow: /*%3D* +Disallow: /
- blocagegazzetta.it bloque OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, Bytespider, DuckAssistBot.
Rang Tranco 1 237 · 12 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -27 +27,28 @@ Disallow: / + +User-agent: bytespider +Disallow: / + +User-agent: chatgpt-user +Disallow: / + +User-agent: oai-searchbot +Disallow: / + +User-agent: claude-user +Disallow: / + +User-agent: claude-searchbot +Disallow: / + +User-agent: perplexitybot +Disallow: / + +User-agent: perplexity-user +Disallow: / + +User-agent: duckassistbot +Disallow: / + +User-agent: mistralai-user +Disallow: /
- partielfiverr.com restreint en partie OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, MistralAI-User, DuckAssistBot.
Rang Tranco 828 · 11 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -56 +56,26 @@ Disallow: /logo-maker/woo + +User-agent: google-extended +User-agent: oai-searchbot +User-agent: claude-searchbot +User-agent: perplexitybot +User-agent: mistralai-index +User-agent: applebot-extended +User-agent: youbot +Allow: / +Disallow: /orders/timeline/* +Disallow: */pinned_flashes/* +Disallow: /gigs/*/share/ +Disallow: /gigs/*/share?* +Disallow: /specials/* +Disallow: /packages/* +Disallow: /categories/silly +Disallow: /categories/fifa +Disallow: /categories/Halloween +Disallow: /categories/Postcards +Disallow: /purchases +Disallow: /user_sessions +Disallow: /users/ +Disallow: /counter/*? +Disallow: /collaborate/* +…
- déblocagecorriere.it débloque Amazonbot.
Rang Tranco 959 · 10 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -20,3 +20,3 @@ User-agent: amazonbot -Disallow: / +Allow: /
- partielllms.txtiporntv.net restreint en partie GPTBot; publie un llms.txt.
Rang Tranco 1 527 · 10 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,15 @@ +User-agent: * +Disallow: /player/float.php +Disallow: /player/html.php +Disallow: /*?action=*$ +Disallow: /*mode=async*$ +Disallow: /*function=get_block*$ +Disallow: /404.php +Disallow: /captcha/* +Disallow: /link/* +Disallow: /related_videos_html/* +Disallow: /feedback/ + +User-agent: gptbot +Disallow: /s/ +Disallow: /get_file/
- blocagepartielcnbc.com bloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; restreint en partie Googlebot.
Rang Tranco 486 · 10 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,77 @@ +User-agent: googlebot +Disallow: /*native-android-mobile +Disallow: /*native-android-tablet +Disallow: /*mobile-native +Disallow: /preview/ +Disallow: /undefined/ +Disallow: /proplayer +Disallow: /appchart/* +Disallow: /search/* +Disallow: /zephr/* +Disallow: /blaize/* + +User-agent: gptbot +User-agent: perplexitybot +User-agent: amazonbot +User-agent: anthropic-ai +User-agent: awariorssbot +User-agent: awariosmartbot +User-agent: bytespider +User-agent: ccbot +User-agent: huggingfacebot +User-agent: chatgpt-user +User-agent: claudebot +User-agent: claude-web +User-agent: dataforseobot +User-agent: diffbot +User-agent: facebookbot +User-agent: google-extended +User-agent: google-notebooklm +User-agent: google-agent +User-agent: magpie-crawler +User-agent: newsnow +User-agent: news-please +User-agent: omgili +User-agent: omgilibot +User-agent: peer39_crawler +User-agent: peer39_crawler/1.0 +User-agent: scrapy +User-agent: turnitinbot …
- déblocageladies.de débloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, CCBot, Bytespider, DuckAssistBot.
Rang Tranco 1 562 · 9 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,2 +1,28 @@ User-agent: * -Disallow: / +Allow: / +Disallow: /*?*umkreis=*adresse=* +Disallow: /*?*suchtext=*adresse=* +Disallow: /*?*umkreis=70* +Disallow: /*?*umkreis=100* +Disallow: /*?*umkreis=150* +Disallow: /*?*umkreis=200* +Disallow: /*?*umkreis=300* +Disallow: /*?*umkreis=70$ +Disallow: /*?*umkreis=100$ +Disallow: /*?*umkreis=150$ +Disallow: /*?*umkreis=200$ +Disallow: /*?*umkreis=300$ +Disallow: /*?page=1$ +Disallow: /*?*page=1$ +Disallow: /*?*page=1&* +Disallow: /*?*&name=* +Disallow: /sex-anzeigen*umkreis=* +Disallow: /cdn-cgi +Disallow: /login +Disallow: /api/affiliates +Disallow: /api/marketplace-banners +Disallow: /api/bookmarks +Disallow: /api/affiliates* +Disallow: /api/marketplace-banners* +Disallow: /api/bookmarks* +Disallow: /*%3D*
- déblocagejustdial.com débloque ClaudeBot.
Rang Tranco 1 341 · 9 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,56 +0,0 @@ -User-agent: * -Disallow: /functions/ -Disallow: /webmain/ -Disallow: /*/wcdata/ -Disallow: /_next/ -Disallow: /web-worker/ -Disallow: /us/ -Disallow: /ca/ -Disallow: /jd-trkr -Disallow: /api/ -Disallow: /jdmart/api/ -Disallow: /real-estate/services/api/ -Disallow: /api/php/appUserId -Disallow: /gapi/ratings -Disallow: /api/india_api_write -Disallow: /api/india_api_read -Disallow: /bestdeals_chatbot/ - -User-agent: oai-searchbot -Allow: / - -User-agent: chatgpt-user -Allow: / - -User-agent: claude-user -Allow: / - -User-agent: claude-searchbot -Allow: / - -User-agent: claudebot -Disallow: / - -User-agent: perplexitybot -Allow: / - -User-agent: perplexity-user -Allow: / - …
- blocagedw.com bloque Perplexity-User.
Rang Tranco 1 390 · 8 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -63 +63,4 @@ Disallow: / + +User-agent: perplexity-user +Disallow: /
- llms.txttarget.com publie un llms.txt.
Rang Tranco 608 · 8 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -0,0 +1,81 @@ +User-agent: * +Disallow: /*/Ntk +Disallow: /*/Ntt +Disallow: /*/Ntx +Disallow: /*%7Cd_ +Disallow: /*/schoollist/ +Disallow: /*BTWN +Disallow: /[path]/ +Disallow: /7078046/ +Disallow: /7079046/ +Disallow: /AddToList +Disallow: /AddToRegistry +Disallow: /admin +Disallow: /advancedGiftRegistrySearchView +Disallow: /AjaxSearchNavigationView +Disallow: /Allons_voter +Disallow: /bp/c/ +Disallow: /bp/guest_mfg_brand +Disallow: /bp/p/ +Disallow: /CallToActionModalView +Disallow: /cgi-bin +Disallow: /cgi-local +Disallow: /Checkout +Disallow: /CheckoutEditItemsDisplayView +Disallow: /CheckoutOrderBillingView +Disallow: /CheckoutOrderShippingView +Disallow: /CheckoutSignInView +Disallow: /co- +Disallow: /common +Disallow: /coupons. +Disallow: /custom-reviews/ +Disallow: /data +Disallow: /database/philboard.mdb +Disallow: /deals/ +Disallow: /dir_on_server/ +Disallow: /EmailCartView +Disallow: /EnlargedImageView +Disallow: /ESPDisplayOptionsViewCmd +Disallow: /ESPModal …
- déblocagepartielcnbc.com débloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider; lève des restrictions partielles pour Googlebot.
Rang Tranco 486 · 8 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,77 +0,0 @@ -User-agent: googlebot -Disallow: /*native-android-mobile -Disallow: /*native-android-tablet -Disallow: /*mobile-native -Disallow: /preview/ -Disallow: /undefined/ -Disallow: /proplayer -Disallow: /appchart/* -Disallow: /search/* -Disallow: /zephr/* -Disallow: /blaize/* - -User-agent: gptbot -User-agent: perplexitybot -User-agent: amazonbot -User-agent: anthropic-ai -User-agent: awariorssbot -User-agent: awariosmartbot -User-agent: bytespider -User-agent: ccbot -User-agent: huggingfacebot -User-agent: chatgpt-user -User-agent: claudebot -User-agent: claude-web -User-agent: dataforseobot -User-agent: diffbot -User-agent: facebookbot -User-agent: google-extended -User-agent: google-notebooklm -User-agent: google-agent -User-agent: magpie-crawler -User-agent: newsnow -User-agent: news-please -User-agent: omgili -User-agent: omgilibot -User-agent: peer39_crawler -User-agent: peer39_crawler/1.0 -User-agent: scrapy -User-agent: turnitinbot …
- déblocagepartielmeraki.com débloque GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot; lève des restrictions partielles pour Googlebot, Bingbot, Applebot, CCBot.
Rang Tranco 176 · 8 novembre 2026diff du robots.txt (groupes concernant l IA)
@@ -1,75 +0,0 @@ -User-agent: 360spider -User-agent: 360spider-image -User-agent: 360spider-video -User-agent: adsbot-google -User-agent: adsbot-google-mobile -User-agent: adsbot-google-mobile-apps -User-agent: adidxbot -User-agent: applebot -User-agent: applenewsbot -User-agent: baiduspider -User-agent: baiduspider-image -User-agent: baiduspider-news -User-agent: baiduspider-video -User-agent: bingbot -User-agent: bingpreview -User-agent: bublupbot -User-agent: ccbot -User-agent: cliqzbot -User-agent: coccoc -User-agent: coccocbot-image -User-agent: coccocbot-web -User-agent: daumoa -User-agent: dazoobot -User-agent: deusu -User-agent: duckduckbot -User-agent: duckduckgo-favicons-bot -User-agent: euripbot -User-agent: exploratodo -User-agent: facebot -User-agent: feedly -User-agent: findxbot -User-agent: googlebot -User-agent: googlebot-image -User-agent: googlebot-mobile -User-agent: googlebot-news -User-agent: googlebot-video -User-agent: haosouspider -User-agent: ichiro -User-agent: istellabot …
Résumés hebdomadaires
- Changements de politique IA, semaine 47 de 2026 (du 16 novembre 2026 au 22 novembre 2026) — 3 changement(s) sur 3 grands domaines : 1 nouveaux blocages de robots d'IA, 1 déblocages, 1 nouveaux llms.txt.
- Changements de politique IA, semaine 46 de 2026 (du 9 novembre 2026 au 15 novembre 2026) — 19 changement(s) sur 16 grands domaines : 5 nouveaux blocages de robots d'IA, 10 déblocages, 1 nouveaux llms.txt.
- Changements de politique IA, semaine 45 de 2026 (du 2 novembre 2026 au 8 novembre 2026) — 4 changement(s) sur 4 grands domaines : 1 nouveaux blocages de robots d'IA, 2 déblocages, 1 nouveaux llms.txt.
Fonctionnement
Deux fois par jour, à heures aléatoires, nous relisons le robots.txt et le /llms.txt des 2 000 domaines les plus populaires de la liste Tranco (1 493 répondent actuellement avec un robots.txt exploitable), avec le User-Agent iabig-policywatch/1.0 (+https://iabig.com/ai-policy-changes/). Chaque fichier est lu avec l'analyseur RFC 9309 de notre testeur de robots.txt pour les 20 robots suivis, avec les définitions du recensement des robots d'IA : bloqué signifie que la page d'accueil et une page quelconque sont interdites, nommément ou via User-agent: * ; partiel signifie que certains chemins sont interdits.
Un changement n'est publié qu'après avoir été observé lors de deux vérifications consécutives, pour que les CDN instables, les pages anti-robots et les erreurs passagères ne créent pas de fausses alertes ; les erreurs serveur, les pages HTML servies à la place de robots.txt et les hôtes injoignables ne modifient jamais l'état enregistré. Nous ne gardons qu'un extrait normalisé des groupes robots.txt qui nomment un robot suivi ou *, et un diff compact — jamais le fichier complet. Les changements marqués « observé entre deux recensements hebdomadaires » viennent de la comparaison des deux derniers recensements, confirmée par notre première vérification.