Summary
From Monday 2 November to Sunday 8 November 2026, iabig.com's server logs recorded 54,256 visits from the crawlers tracked by the observatory (spoofed visits excluded), +51% compared with the previous week (35,837).
The most active crawler was Googlebot (Google) with 50,922 visits, 94% of the total, followed by GPTBot (2,850) and Amazonbot (144).
By company, Google accounted for 94% of visits, ahead of OpenAI (5%) and Amazon (0%).
Seen on iabig.com for the first time: Applebot.
499 other requests used a known crawler's name without coming from its official IP ranges (spoofed); they are left out of the figures.
The most crawled part of the site was “Other URLs (old addresses, 404 errors…)” (95% of visits with a known section).
Visits per day
Most active crawlers
| Crawler | Visits | Prev. week | Change | IP-verified |
|---|---|---|---|---|
| Googlebot Google · Search | 50,922 | 32,685 | +56% | 50,922 |
| GPTBot OpenAI · Training | 2,850 | 2,482 | +15% | 2,850 |
| Amazonbot Amazon · Training | 144 | 196 | −27% | 0 |
| Bingbot Microsoft · Search | 120 | 59 | +103% | 120 |
| Applebot Apple · Search | 58 | 0 | new | 58 |
| OAI-SearchBot OpenAI · Search | 40 | 49 | −18% | 40 |
| Bytespider ByteDance · Training | 34 | 13 | +162% | 0 |
| ClaudeBot Anthropic · Training | 31 | 196 | −84% | 31 |
| meta-externalagent Meta · Training | 31 | 145 | −79% | 0 |
| meta-externalfetcher Meta · Browsing | 25 | 4 | +525% | 0 |
| ChatGPT-User OpenAI · Browsing | 1 | 8 | −88% | 1 |
Share by company
| 94% · 50,922 | |
| OpenAI | 5% · 2,891 |
| Amazon | 0% · 144 |
| Microsoft | 0% · 120 |
| Apple | 0% · 58 |
| Meta | 0% · 56 |
| ByteDance | 0% · 34 |
| Anthropic | 0% · 31 |
Most crawled sections of the site
| Other URLs (old addresses, 404 errors…) | 51,644 |
| robots.txt | 2,240 |
| Guides | 116 |
| AI crawler profiles | 73 |
| Tools | 45 |
| Technical files (images, scripts, .well-known) | 28 |
| Checker (home page) | 27 |
| Sitemap and feed | 24 |
| Blocklists | 18 |
| About, privacy, legal | 17 |
| Observatory and reports | 16 |
| API | 5 |
| llms.txt | 3 |
Elsewhere on the web this week
Our AI policy change tracker recorded 4 robots.txt or llms.txt change(s) on 4 of the 2,000 most popular websites; most newly blocked crawlers: Perplexity-User. This week's digest.
- meraki.com (#176) : unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, meta-externalfetcher, Amazonbot, MistralAI-User, Bytespider, DuckAssistBot
- cnbc.com (#486) : unblocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider
- target.com (#608) : published an llms.txt
- dw.com (#1390) : blocked Perplexity-User
AI-Readiness Index: median score 77.0/100 across 4,564 home pages of the top 10,000; 11% are JavaScript shells, 46% publish JSON-LD.
Method
Crawlers are identified by User-Agent (list in the observatory), then verified against their operator's official IP ranges. Spoofed visits are excluded; “unverifiable” means the operator publishes no IP ranges. Googlebot, Bingbot and Applebot are counted because their indexes also feed AI answers. Only totals per crawler, per day and per site section are kept: no IP address and nothing about human visitors. A report is frozen once published (unless data for that week arrives late).