What the audit checks
- AI crawler access: robots.txt rules for OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and the AI training crawlers, page by page (path rules included).
- Readable without JavaScript: pages that are empty shells for crawlers that do not run JavaScript, thin pages, very heavy HTML.
- Understanding: titles, meta descriptions, a single H1, canonical URLs, language, Open Graph, JSON-LD structured data (including invalid blocks), image alt text.
- Indexing and discovery: noindex pages, linked pages in error, sitemap, llms.txt, near-duplicate pages, bot-challenge pages, Cloudflare.
- Security: hidden instructions aimed at AI assistants (prompt injection in hidden text, comments or invisible characters).
Fixes are sorted by priority — impact on AI visibility × share of pages affected — and link to the relevant guide. Your home page is also scored on the rubric of the AI-Readiness Index, so you can compare it with the 10,000 most popular websites.
How we crawl
Our crawler identifies itself as iabig-audit/1.0 (+https://iabig.com/tools/site-audit/), honours robots.txt (token iabig-audit), stays on your site, pauses between requests and stops after 30 pages or 3 minutes. Pages are analysed in memory: we keep only the report, never the content of your pages. Reports are not listed or indexed and are deleted after 30 days.
More free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
Free uptime monitor
Get alerted when your website goes down, its HTTPS certificate is about to expire, or ChatGPT, Claude and Perplexity can no longer read it. 10 sites free, email, Discord, Slack and ntfy alerts.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
Verify an AI crawler IP
Check whether an IP address really belongs to OpenAI, Anthropic, Perplexity, Google, Bing, Apple or another AI company using their official IP lists.
Free llms.txt generator from your sitemap
Generate a draft llms.txt in the llmstxt.org format from your website's sitemap - titles, descriptions and sections ready to review.
Free prompt injection detector
Check whether a web page or a text hides instructions aimed at AI assistants - hidden text, HTML comments and invisible Unicode characters.
AI API cost calculator
Compare what 40 AI models (OpenAI, Anthropic, Google, Mistral, DeepSeek, xAI) really cost for your volumes: verified official prices, caching, batch and long-context tiers.