Summary
From Monday 26 October to Sunday 1 November 2026, iabig.com's server logs recorded 35,837 visits from the crawlers tracked by the observatory (spoofed visits excluded), +280% compared with the previous week (9,434). The previous week was only partly covered (data collection started then), so the change is overstated.
The most active crawler was Googlebot (Google) with 32,685 visits, 91% of the total, followed by GPTBot (2,482) and ClaudeBot (196).
By company, Google accounted for 91% of visits, ahead of OpenAI (7%) and Amazon (1%).
Seen on iabig.com for the first time: Amazonbot, Bytespider, ChatGPT-User, meta-externalagent, meta-externalfetcher.
203 other requests used a known crawler's name without coming from its official IP ranges (spoofed); they are left out of the figures.
The most crawled part of the site was “Other URLs (old addresses, 404 errors…)” (94% of visits with a known section).
Visits per day
Most active crawlers
| Crawler | Visits | Prev. week | Change | IP-verified |
|---|---|---|---|---|
| Googlebot Google · Search | 32,685 | 8,625 | +279% | 32,685 |
| GPTBot OpenAI · Training | 2,482 | 506 | +391% | 2,482 |
| ClaudeBot Anthropic · Training | 196 | 291 | −33% | 196 |
| Amazonbot Amazon · Training | 196 | 0 | new | 0 |
| meta-externalagent Meta · Training | 145 | 0 | new | 0 |
| Bingbot Microsoft · Search | 59 | 8 | +638% | 59 |
| OAI-SearchBot OpenAI · Search | 49 | 4 | +1125% | 49 |
| Bytespider ByteDance · Training | 13 | 0 | new | 0 |
| ChatGPT-User OpenAI · Browsing | 8 | 0 | new | 8 |
| meta-externalfetcher Meta · Browsing | 4 | 0 | new | 0 |
Share by company
| 91% · 32,685 | |
| OpenAI | 7% · 2,539 |
| Amazon | 1% · 196 |
| Anthropic | 1% · 196 |
| Meta | 0% · 149 |
| Microsoft | 0% · 59 |
| ByteDance | 0% · 13 |
Most crawled sections of the site
Sections tracked from 28 Oct 2026.
| Other URLs (old addresses, 404 errors…) | 22,345 |
| robots.txt | 1,143 |
| AI crawler profiles | 101 |
| Guides | 78 |
| Sitemap and feed | 58 |
| Tools | 29 |
| Blocklists | 27 |
| Checker (home page) | 20 |
| Observatory and reports | 18 |
| About, privacy, legal | 7 |
| API | 2 |
| llms.txt | 1 |
| Technical files (images, scripts, .well-known) | 1 |
Method
Crawlers are identified by User-Agent (list in the observatory), then verified against their operator's official IP ranges. Spoofed visits are excluded; “unverifiable” means the operator publishes no IP ranges. Googlebot, Bingbot and Applebot are counted because their indexes also feed AI answers. Only totals per crawler, per day and per site section are kept: no IP address and nothing about human visitors. A report is frozen once published (unless data for that week arrives late).