How the test works

The tester applies the official standard, RFC 9309, which Google, OpenAI, Anthropic and other compliant crawlers follow:

a crawler follows the User-agent group with its exact name (case-insensitive); if there is none, it follows the User-agent: * group;
several groups with the same name are merged;
the longest matching Allow or Disallow rule wins; on a tie, Allow wins;
the * wildcard (any characters) and $ (end of URL) are supported;
an empty Disallow: line disallows nothing.

Common mistakes

Assuming User-agent: * adds up with a crawler's own group: once a group names the crawler, the * group is ignored for it. Copy your general disallows into each named group.
Blocking Google-Extended thinking it blocks Googlebot (or the reverse): they are two different tokens with different effects.
Forgetting that a robots.txt returning a 5xx error is treated as a full disallow.

To check the robots.txt your site actually serves, use the AI visibility checker.

Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.

Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.

Check whether an IP address really belongs to OpenAI, Anthropic, Perplexity, Google, Bing, Apple or another AI company using their official IP lists.

Generate a draft llms.txt in the llmstxt.org format from your website's sitemap - titles, descriptions and sections ready to review.

Check whether a web page or a text hides instructions aimed at AI assistants - hidden text, HTML comments and invisible Unicode characters.