IABIGFR · Français
Weekly report

AI crawlers on iabig.com: week 46, 2026

What AI crawlers did on iabig.com, 9 Nov – 15 Nov 2026.

Published 16 Nov 2026. Generated automatically from the server logs.

One website only. These figures describe what AI crawlers did on iabig.com this week, according to its own server logs. They do not measure the web as a whole: a crawler's activity here depends on this site's size, its links and its robots.txt (which allows every crawler), and says nothing about what it does elsewhere.

Summary

From Monday 9 November to Sunday 15 November 2026, iabig.com's server logs recorded 44,967 visits from the crawlers tracked by the observatory (spoofed visits excluded), −17% compared with the previous week (54,256).

The most active crawler was Googlebot (Google) with 41,175 visits, 92% of the total, followed by GPTBot (2,833) and Amazonbot (365).

By company, Google accounted for 92% of visits, ahead of OpenAI (7%) and Amazon (1%).

Seen on iabig.com for the first time: PerplexityBot.

164 other requests used a known crawler's name without coming from its official IP ranges (spoofed); they are left out of the figures.

The most crawled part of the site was “Other URLs (old addresses, 404 errors…)” (94% of visits with a known section).

Visits per day

Monday 9 November 2026: 8,8778,877Mon 9Tuesday 10 November 2026: 4,9694,969Tue 10Wednesday 11 November 2026: 4,7554,755Wed 11Thursday 12 November 2026: 4,4474,447Thu 12Friday 13 November 2026: 7,9047,904Fri 13Saturday 14 November 2026: 8,4848,484Sat 14Sunday 15 November 2026: 5,5315,531Sun 15

Most active crawlers

CrawlerVisitsPrev. weekChangeIP-verified
Googlebot
Google · Search
41,17550,922−19%41,175
GPTBot
OpenAI · Training
2,8332,850−1%2,833
Amazonbot
Amazon · Training
365144+153%0
Bingbot
Microsoft · Search
241120+101%241
Applebot
Apple · Search
20658+255%206
OAI-SearchBot
OpenAI · Search
8440+110%84
Bytespider
ByteDance · Training
1734−50%0
meta-externalagent
Meta · Training
1331−58%0
ChatGPT-User
OpenAI · Browsing
131+1200%13
meta-externalfetcher
Meta · Browsing
1225−52%0
PerplexityBot
Perplexity · Search
80new8
ClaudeBot
Anthropic · Training
031−100%0

Share by company

Google92% · 41,175
OpenAI7% · 2,930
Amazon1% · 365
Microsoft1% · 241
Apple0% · 206
Meta0% · 25
ByteDance0% · 17
Perplexity0% · 8

Most crawled sections of the site

Other URLs (old addresses, 404 errors…)42,275
robots.txt2,149
Guides201
AI crawler profiles131
Tools61
Checker (home page)43
Observatory and reports29
Technical files (images, scripts, .well-known)25
Blocklists19
About, privacy, legal13
Sitemap and feed11
API6
llms.txt4

Elsewhere on the web this week

Our AI policy change tracker recorded 19 robots.txt or llms.txt change(s) on 16 of the 2,000 most popular websites; most newly blocked crawlers: ClaudeBot, Bytespider, GPTBot, Applebot-Extended. This week's digest.

  • cnbc.com (#486) : blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider
  • unesco.org (#614) : removed its llms.txt
  • techcrunch.com (#633) : unblocked ChatGPT-User, Google-Extended
  • fiverr.com (#828) :
  • hbr.org (#912) : blocked ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider

AI-Readiness Index: median score 77/100 across 5,517 home pages of the top 10,000; 12% are JavaScript shells, 47% publish JSON-LD.

Method

Crawlers are identified by User-Agent (list in the observatory), then verified against their operator's official IP ranges. Spoofed visits are excluded; “unverifiable” means the operator publishes no IP ranges. Googlebot, Bingbot and Applebot are counted because their indexes also feed AI answers. Only totals per crawler, per day and per site section are kept: no IP address and nothing about human visitors. A report is frozen once published (unless data for that week arrives late).

Can AI crawlers read your own website? Run the free checker.