robots.txt tester for AI crawlers

Paste a robots.txt and a page path: the tester shows, for every AI crawler, whether it may access the page and which rule applies.

robots.txt content Page to test (path or full URL) Test

How the test works

The tester applies the official standard, RFC 9309, which Google, OpenAI, Anthropic and other compliant crawlers follow:

• a crawler follows the User-agent group with its exact name (case-insensitive); if there is none, it follows the User-agent: * group;
• several groups with the same name are merged;
• the longest matching Allow or Disallow rule wins; on a tie, Allow wins;
• the * wildcard (any characters) and $ (end of URL) are supported;
• an empty Disallow: line disallows nothing.

Common mistakes

• Assuming User-agent: * adds up with a crawler's own group: once a group names the crawler, the * group is ignored for it. Copy your general disallows into each named group.
• Blocking Google-Extended thinking it blocks Googlebot (or the reverse): they are two different tokens with different effects.
• Forgetting that a robots.txt returning a 5xx error is treated as a full disallow.

To check the robots.txt your site actually serves, use the AI visibility checker.

More free tools