IABIGFR · Français
Weekly report

AI crawlers on iabig.com: week 45, 2026

What AI crawlers did on iabig.com, 2 Nov – 8 Nov 2026.

Published 9 Nov 2026. Generated automatically from the server logs.

One website only. These figures describe what AI crawlers did on iabig.com this week, according to its own server logs. They do not measure the web as a whole: a crawler's activity here depends on this site's size, its links and its robots.txt (which allows every crawler), and says nothing about what it does elsewhere.

Summary

From Monday 2 November to Sunday 8 November 2026, iabig.com's server logs recorded 54,256 visits from the crawlers tracked by the observatory (spoofed visits excluded), +51% compared with the previous week (35,837).

The most active crawler was Googlebot (Google) with 50,922 visits, 94% of the total, followed by GPTBot (2,850) and Amazonbot (144).

By company, Google accounted for 94% of visits, ahead of OpenAI (5%) and Amazon (0%).

Seen on iabig.com for the first time: Applebot.

499 other requests used a known crawler's name without coming from its official IP ranges (spoofed); they are left out of the figures.

The most crawled part of the site was “Other URLs (old addresses, 404 errors…)” (95% of visits with a known section).

Visits per day

Monday 2 November 2026: 5,7225,722Mon 2Tuesday 3 November 2026: 7,4517,451Tue 3Wednesday 4 November 2026: 7,7867,786Wed 4Thursday 5 November 2026: 9,7429,742Thu 5Friday 6 November 2026: 8,1368,136Fri 6Saturday 7 November 2026: 9,6509,650Sat 7Sunday 8 November 2026: 5,7695,769Sun 8

Most active crawlers

CrawlerVisitsPrev. weekChangeIP-verified
Googlebot
Google · Search
50,92232,685+56%50,922
GPTBot
OpenAI · Training
2,8502,482+15%2,850
Amazonbot
Amazon · Training
144196−27%0
Bingbot
Microsoft · Search
12059+103%120
Applebot
Apple · Search
580new58
OAI-SearchBot
OpenAI · Search
4049−18%40
Bytespider
ByteDance · Training
3413+162%0
ClaudeBot
Anthropic · Training
31196−84%31
meta-externalagent
Meta · Training
31145−79%0
meta-externalfetcher
Meta · Browsing
254+525%0
ChatGPT-User
OpenAI · Browsing
18−88%1

Share by company

Google94% · 50,922
OpenAI5% · 2,891
Amazon0% · 144
Microsoft0% · 120
Apple0% · 58
Meta0% · 56
ByteDance0% · 34
Anthropic0% · 31

Most crawled sections of the site

Other URLs (old addresses, 404 errors…)51,644
robots.txt2,240
Guides116
AI crawler profiles73
Tools45
Technical files (images, scripts, .well-known)28
Checker (home page)27
Sitemap and feed24
Blocklists18
About, privacy, legal17
Observatory and reports16
API5
llms.txt3

Elsewhere on the web this week

Our AI policy change tracker recorded 4 robots.txt or llms.txt change(s) on 4 of the 2,000 most popular websites; most newly blocked crawlers: Perplexity-User. This week's digest.

  • meraki.com (#176) : unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot
  • cnbc.com (#486) : unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider
  • target.com (#608) : published an llms.txt
  • dw.com (#1390) : blocked Perplexity-User

AI-Readiness Index: median score 77.0/100 across 4,564 home pages of the top 10,000; 11% are JavaScript shells, 46% publish JSON-LD.

Method

Crawlers are identified by User-Agent (list in the observatory), then verified against their operator's official IP ranges. Spoofed visits are excluded; “unverifiable” means the operator publishes no IP ranges. Googlebot, Bingbot and Applebot are counted because their indexes also feed AI answers. Only totals per crawler, per day and per site section are kept: no IP address and nothing about human visitors. A report is frozen once published (unless data for that week arrives late).

Can AI crawlers read your own website? Run the free checker.