Summary
From Monday 9 November to Sunday 15 November 2026, iabig.com's server logs recorded 44,967 visits from the crawlers tracked by the observatory (spoofed visits excluded), −17% compared with the previous week (54,256).
The most active crawler was Googlebot (Google) with 41,175 visits, 92% of the total, followed by GPTBot (2,833) and Amazonbot (365).
By company, Google accounted for 92% of visits, ahead of OpenAI (7%) and Amazon (1%).
Seen on iabig.com for the first time: PerplexityBot.
164 other requests used a known crawler's name without coming from its official IP ranges (spoofed); they are left out of the figures.
The most crawled part of the site was “Other URLs (old addresses, 404 errors…)” (94% of visits with a known section).
Visits per day
Most active crawlers
| Crawler | Visits | Prev. week | Change | IP-verified |
|---|---|---|---|---|
| Googlebot Google · Search | 41,175 | 50,922 | −19% | 41,175 |
| GPTBot OpenAI · Training | 2,833 | 2,850 | −1% | 2,833 |
| Amazonbot Amazon · Training | 365 | 144 | +153% | 0 |
| Bingbot Microsoft · Search | 241 | 120 | +101% | 241 |
| Applebot Apple · Search | 206 | 58 | +255% | 206 |
| OAI-SearchBot OpenAI · Search | 84 | 40 | +110% | 84 |
| Bytespider ByteDance · Training | 17 | 34 | −50% | 0 |
| meta-externalagent Meta · Training | 13 | 31 | −58% | 0 |
| ChatGPT-User OpenAI · Browsing | 13 | 1 | +1200% | 13 |
| meta-externalfetcher Meta · Browsing | 12 | 25 | −52% | 0 |
| PerplexityBot Perplexity · Search | 8 | 0 | new | 8 |
| ClaudeBot Anthropic · Training | 0 | 31 | −100% | 0 |
Share by company
| 92% · 41,175 | |
| OpenAI | 7% · 2,930 |
| Amazon | 1% · 365 |
| Microsoft | 1% · 241 |
| Apple | 0% · 206 |
| Meta | 0% · 25 |
| ByteDance | 0% · 17 |
| Perplexity | 0% · 8 |
Most crawled sections of the site
| Other URLs (old addresses, 404 errors…) | 42,275 |
| robots.txt | 2,149 |
| Guides | 201 |
| AI crawler profiles | 131 |
| Tools | 61 |
| Checker (home page) | 43 |
| Observatory and reports | 29 |
| Technical files (images, scripts, .well-known) | 25 |
| Blocklists | 19 |
| About, privacy, legal | 13 |
| Sitemap and feed | 11 |
| API | 6 |
| llms.txt | 4 |
Elsewhere on the web this week
Our AI policy change tracker recorded 19 robots.txt or llms.txt change(s) on 16 of the 2,000 most popular websites; most newly blocked crawlers: ClaudeBot, Bytespider, GPTBot, Applebot-Extended. This week's digest.
- cnbc.com (#486) : blocked GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider
- unesco.org (#614) : removed its llms.txt
- techcrunch.com (#633) : unblocked ChatGPT-User, Google-Extended
- fiverr.com (#828) :
- hbr.org (#912) : blocked ClaudeBot, Applebot-Extended, meta-externalagent, meta-externalfetcher, Bytespider
AI-Readiness Index: median score 77/100 across 5,517 home pages of the top 10,000; 12% are JavaScript shells, 47% publish JSON-LD.
Method
Crawlers are identified by User-Agent (list in the observatory), then verified against their operator's official IP ranges. Spoofed visits are excluded; “unverifiable” means the operator publishes no IP ranges. Googlebot, Bingbot and Applebot are counted because their indexes also feed AI answers. Only totals per crawler, per day and per site section are kept: no IP address and nothing about human visitors. A report is frozen once published (unless data for that week arrives late).