How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccess
Count visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
Analytics tools built on a JavaScript snippet only see visitors that run the snippet. AI crawlers that do not execute it, such as CCBot according to Common Crawl, never show up there. To know who actually reads your site, which pages and how often, the reliable source is still the web server's access log. This guide starts from the standard log format, gives the useful grep, awk and GoAccess commands, then a complete script that produces a per-crawler summary.
The log format you need to know
nginx uses the "combined" format by default, and Apache provides it under the same name. A line looks like this:
20.171.206.5 - - [24/Oct/2026:10:00:01 +0200] "GET /guide/ HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
In nginx, this format is predefined under the name combined, and it is the one access_log uses when no format is specified:
# Built into nginx (no need to copy it)
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
In Apache, the format has the same name. Make sure your CustomLog actually uses it, and not common, which does not record the User-Agent:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
CustomLog ${APACHE_LOG_DIR}/access.log combined
Usual locations are /var/log/nginx/access.log for nginx, /var/log/apache2/access.log on Debian and Ubuntu, and /var/log/httpd/access_log on Red Hat-based distributions. Older logs are often compressed (access.log.2.gz), and zcat -f reads compressed and uncompressed files alike.
Splitting on the double quote is the key to the awk commands that follow. With awk -F'"', field $2 holds the request (GET /guide/ HTTP/1.1), $3 the status code and size, and $6 the User-Agent.
Crawlers to look for
The tokens below are the ones published by the operators. They appear in the User-Agent header of requests, except for Google-Extended and Applebot-Extended, which are only robots.txt tokens. You will never see them in a log.
| Family | Tokens to search for |
|---|---|
| Training | GPTBot, ClaudeBot, CCBot, meta-externalagent, MistralAI-Training, Amazonbot, Bytespider |
| Search and indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amzn-SearchBot, meta-webindexer, MistralAI-Index, DuckAssistBot |
| User-initiated | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, meta-externalfetcher, Amzn-User, Google-Agent |
Version numbers change (GPTBot/1.4 today), so always search for the token, never the full string.
Quick commands
LOG=/var/log/nginx/access.log
BOTS='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-externalagent|meta-externalfetcher|meta-webindexer|Applebot|Amazonbot|Amzn-SearchBot|Amzn-User|MistralAI-User|MistralAI-Index|MistralAI-Training|DuckAssistBot|Google-Agent'
# Total number of AI crawler requests
grep -ciE "$BOTS" "$LOG"
# Requests per crawler, archived logs included
# (grep -o would also count the documentation URLs inside the User-Agent,
# so only the first match in the User-Agent field is kept; requires GNU awk)
zcat -f /var/log/nginx/access.log* \
| awk -F'"' -v bots="$BOTS" 'BEGIN { IGNORECASE = 1 } match($6, bots) { print substr($6, RSTART, RLENGTH) }' \
| sort | uniq -c | sort -rn
# Top 20 pages read by GPTBot (URL parameters stripped)
awk -F'"' 'tolower($6) ~ /gptbot/ { split($2, r, " "); sub(/\?.*/, "", r[2]); print r[2] }' "$LOG" \
| sort | uniq -c | sort -rn | head -20
# Status codes returned to ClaudeBot (spot 403, 404 and 5xx)
awk -F'"' '$6 ~ /ClaudeBot/ { split($3, s, " "); print s[1] }' "$LOG" | sort | uniq -c | sort -rn
# Requests per day for OAI-SearchBot
awk -F'"' '$6 ~ /OAI-SearchBot/ { print substr($1, index($1, "[") + 1, 11) }' "$LOG" | uniq -c
# Who reads robots.txt, and with which status code
awk -F'"' '$2 ~ /^GET \/robots\.txt/ { split($3, s, " "); print s[1], $6 }' "$LOG" \
| grep -iE "$BOTS" | sort | uniq -c | sort -rn
# Bytes transferred per crawler
awk -F'"' -v bots="$BOTS" 'BEGIN { IGNORECASE = 1 }
match($6, bots) { split($3, s, " "); o[substr($6, RSTART, RLENGTH)] += s[2] }
END { for (b in o) printf "%12d %s\n", o[b], b }' "$LOG" | sort -rn
The commands that use IGNORECASE require GNU awk (gawk). With another awk, remove the BEGIN block and match the tokens' case exactly.
The robots.txt command is useful for checking your setup. Under RFC 9309, a crawler that gets a 5xx error on that file must treat the whole site as disallowed.
Writing AI crawlers to a dedicated log
Rather than filtering after the fact, you can have the server write these requests to a separate file.
With nginx, in the http block:
map $http_user_agent $ai_bot {
default 0;
"~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-external|meta-webindexer|Amazonbot|Amzn-|MistralAI-|DuckAssistBot|Google-Agent|Applebot)" 1;
}
Then in the server block:
# access_log in server replaces the inherited directives, so the main log is kept.
access_log /var/log/nginx/access.log combined;
access_log /var/log/nginx/ai-bots.log combined if=$ai_bot;
With Apache (mod_setenvif and mod_log_config modules):
SetEnvIfNoCase User-Agent "(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-external|meta-webindexer|Amazonbot|Amzn-|MistralAI-|DuckAssistBot|Google-Agent|Applebot)" ai_bot
CustomLog ${APACHE_LOG_DIR}/ai-bots.log combined env=ai_bot
Remember to add this file to your logrotate configuration, like the server's other logs.
GoAccess
GoAccess parses logs in the combined format and produces an HTML report. Since version 1.12 (September 2026), it puts AI crawlers in a separate "AI Crawlers" category, distinct from traditional search engines.
# Full report
goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html
# Report limited to AI crawlers, compressed logs included
zcat -f /var/log/nginx/access.log* \
| grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-external' \
| goaccess - --log-format=COMBINED -o ai-bots.html
With an older version, declare the tokens yourself in a list passed with --browsers-file (one word, one or more tabs, then the category):
printf '%s\t\tCrawlers\n' GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User > ai.list
goaccess /var/log/nginx/access.log --log-format=COMBINED --browsers-file=ai.list -o report.html
If your server writes the virtual host at the start of each line, use VCOMBINED instead of COMBINED.
Ready-to-use script
This script reads one or more logs in the combined format, compressed or not, and prints the number of requests per crawler, status codes, the number of distinct IP addresses, activity per day and the most-read pages.
#!/usr/bin/env bash
# ai-bot-logs.sh - summary of AI crawler visits ("combined" format)
# Usage: ./ai-bot-logs.sh /var/log/nginx/access.log* (.gz files accepted)
set -euo pipefail
[ "$#" -ge 1 ] || { echo "usage: $0 log_file [log_file...]" >&2; exit 1; }
# Tokens searched for in the User-Agent (case-insensitive)
BOTS="GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User Google-Agent Applebot Amazonbot Amzn-SearchBot Amzn-User meta-externalagent meta-externalfetcher meta-webindexer CCBot Bytespider MistralAI-User MistralAI-Index MistralAI-Training DuckAssistBot"
zcat -f -- "$@" | awk -F'"' -v bots="$BOTS" '
BEGIN { n = split(bots, B, " "); for (i = 1; i <= n; i++) L[i] = tolower(B[i]) }
{
ua = tolower($6); bot = ""
for (i = 1; i <= n; i++) if (index(ua, L[i])) { bot = B[i]; break }
if (bot == "") next
split($2, req, " "); url = req[2]; sub(/\?.*/, "", url) # path without parameters
split($3, st, " "); code = st[1]
split($1, f, "["); day = substr(f[2], 1, 11) # 25/Oct/2026
split($1, h, " "); ip = h[1]
hits[bot]++; codes[bot " " code]++; pages[bot "\t" url]++
days[day " " bot]++; ips[bot " " ip] = 1
}
END {
print "== Requests per crawler"
for (b in hits) printf "%8d %s\n", hits[b], b | "sort -rn"; close("sort -rn")
print "\n== Status codes per crawler"
for (k in codes) printf "%8d %s\n", codes[k], k | "sort -k2,2 -k1,1rn"; close("sort -k2,2 -k1,1rn")
print "\n== Distinct IP addresses per crawler"
for (k in ips) { split(k, a, " "); nip[a[1]]++ }
for (b in nip) printf "%8d %s\n", nip[b], b | "sort -rn"; close("sort -rn")
print "\n== Requests per day"
for (k in days) printf "%8d %s\n", days[k], k | "sort -k2,2 -k1,1rn"; close("sort -k2,2 -k1,1rn")
print "\n== Top 20 pages (all AI crawlers)"
for (k in pages) { split(k, a, "\t"); tot[a[2]] += pages[k] }
for (u in tot) printf "%8d %s\n", tot[u], u | "sort -rn | head -20"; close("sort -rn | head -20")
}'
chmod +x ai-bot-logs.sh
./ai-bot-logs.sh /var/log/nginx/access.log /var/log/nginx/access.log.*.gz
sudo ./ai-bot-logs.sh /var/log/apache2/access.log*
Sample output on a small test log:
== Requests per crawler
3 GPTBot
1 PerplexityBot
1 ClaudeBot
1 Claude-SearchBot
== Status codes per crawler
1 Claude-SearchBot 200
1 ClaudeBot 200
1 GPTBot 200
1 GPTBot 403
1 GPTBot 404
1 PerplexityBot 200
The script counts what the requests claim to be. Dates are sorted as text, which works for a single month. For a longer period, export the data to a spreadsheet or a database.
Breaking results down crawler by crawler
The overall summary mixes crawlers with very different roles. To compare what each crawler reads, a loop is enough. It produces a page ranking per crawler, which shows whether a search crawler actually reaches your important content or wears itself out on filter, pagination or archive pages.
for bot in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot; do
echo "== $bot"
zcat -f /var/log/nginx/access.log* \
| awk -F'"' -v b="$bot" 'index(tolower($6), tolower(b)) { split($2, r, " "); sub(/\?.*/, "", r[2]); print r[2] }' \
| sort | uniq -c | sort -rn | head -10
done
Tracking trends over time
Logs are often deleted after a few weeks by logrotate. To keep a record, save a per-crawler count to a CSV file every night, from the previous day's log (here access.log.1, the usual name after daily rotation):
# /etc/cron.d/ai-bots - every day at 00:30
30 0 * * * root gawk -F'"' 'BEGIN { IGNORECASE = 1; d = strftime("\%Y-\%m-\%d", systime() - 86400) } match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot/) { n[substr($6, RSTART, RLENGTH)]++ } END { for (b in n) print d ";" b ";" n[b] }' /var/log/nginx/access.log.1 >> /var/log/ai-bots.csv
In a cron file, the % character must be escaped, hence the \%. The strftime and systime functions and IGNORECASE are specific to GNU awk, which is why the line calls gawk explicitly (on Debian and Ubuntu, the default awk is often mawk). The resulting file opens in any spreadsheet, with one column for the date, one for the crawler and one for the request count.
What should you look at first? Three signals are usually enough. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) should show up regularly if you allow them. If they stay absent, check robots.txt and the firewall. User-initiated agents (ChatGPT-User, Claude-User, Perplexity-User) show which pages are read at the moment a user asks a question. Finally, a crawler you have disallowed in robots.txt that keeps reading pages other than /robots.txt needs checking. Either it is an impostor, or the rule does not target it correctly.
Read the numbers with caution
- Spoofing: a User-Agent is easy to fake. An IP that claims to be GPTBot but is not in OpenAI's ranges is not GPTBot. Cross-check the addresses against the official JSON files, as explained in How to verify GPTBot, ClaudeBot and PerplexityBot.
- CDNs and proxies: behind a CDN, the first column holds the proxy's address. Enable nginx's
realipmodule or Apache'smod_remoteipto log the client IP, or use the CDN's logs. - Shared crawls: OpenAI says a single crawl can serve both
GPTBotandOAI-SearchBotwhen both are allowed. The absence of one does not mean your pages are being ignored. - On-demand visits: a
ChatGPT-User,Claude-UserorPerplexity-Userrequest usually corresponds to a user's question. It is the closest indicator of an assistant actually reading your content, but it says nothing about how many people will see the answer. - Error codes: a high rate of 403 or 429 for a search crawler often means a firewall or rate limit is blocking it without your knowledge.
To check what your robots.txt and pages tell these crawlers, use the IABIG checker.
Official sources
- nginx, logging module: https://nginx.org/en/docs/http/ngx_http_log_module.html
- nginx, map module: https://nginx.org/en/docs/http/ngx_http_map_module.html
- Apache, mod_log_config: https://httpd.apache.org/docs/2.4/mod/mod_log_config.html
- Apache, mod_setenvif: https://httpd.apache.org/docs/2.4/mod/mod_setenvif.html
- GoAccess, manual and 1.12 release notes: https://goaccess.io/man and https://github.com/allinurl/goaccess/blob/master/ChangeLog
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Common Crawl, FAQ (JavaScript execution): https://commoncrawl.org/faq
- Google, Google-Extended: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google, user-triggered fetchers (Google-Agent): https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
- Apple, About Applebot: https://support.apple.com/en-us/119829
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and moreThe robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNSConfirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers comparedWhat GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.
- ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers comparedWhat ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.