Block or Allow AI Crawlers on Shopify - Customize robots.txt.liquid
Add groups to robots.txt.liquid that block GPTBot or ClaudeBot while keeping your store visible in ChatGPT, Claude and Perplexity, plus the limits to know.
Shopify generates every store's robots.txt file automatically and keeps it updated as the platform evolves. There is no file to upload: to add your own rules, you create a theme template named robots.txt.liquid. This guide shows how to add groups for AI crawlers without losing Shopify's default rules, explains what this file does not control (in particular, how your catalog reaches shopping assistants), and how to check the result.
What Shopify allows and what it doesn't
The file is served at the root of the store's primary domain, for example https://www.example.com/robots.txt. According to the Help Center, customization lets you allow or disallow URLs, add Crawl-delay rules for specific crawlers, add extra sitemaps, and block certain crawlers. It works exclusively through the robots.txt.liquid template, which can only use six Liquid objects: robots, group, rule, user_agent, sitemap and request.
Shopify warns that this is an unsupported customization: Shopify Support won't help you edit the file, and a mistake can cost you all your search engine traffic. The Help Center also asks you to remove any earlier workaround, for example one built on Cloudflare, before editing the template.
According to Shopify's help page, two layers of control apply to AI crawlers:
- The network layer, where Shopify handles bot management itself. Nothing is expected from you, and Shopify does not recommend putting a proxy in front of your store.
- The
robots.txtlayer, which you control with the template. Shopify calls these rules "directional and advisory": not every crawler respects them.
Blocking crawlers doesn't turn off agentic storefronts
This is what sets Shopify apart. If you sell through agentic storefronts such as ChatGPT or Microsoft Copilot, your product data is sent to them by Shopify Catalog, independently of robots.txt. Blocking an AI crawler in the file only affects discovery through the open web.
This syndication is managed in the Shopify admin, under Sales channels > Agentic. The Allow Shopify to manage for me setting is on by default; once you turn it off, you can deactivate Shopify Catalog access for all third-party storefronts at once or channel by channel. Shopify says it can take up to 7 days for sharing to stop, and that your products can still be discovered through web crawling. Two exceptions are worth knowing: Google AI Mode and Gemini are fed through the Google & YouTube channel or Merchant Center rather than this page, and Shopify's own Shop app has no toggle here (Shopify points to the Unlisted product status or the seo.hidden metafield to hide products everywhere). The two levers are therefore complementary: robots.txt for the crawlers that read your pages, the agentic settings for the catalog feed.
Create the robots.txt.liquid template
In the Shopify admin:
- Go to Online Store.
- For the relevant theme, click the … button, then Edit code.
- Click Add a new template, then select robots.
- Click Create template.
- Replace the content with the example below, then save the file in your published theme.
Before anything else, save a copy of your current robots.txt: the developer documentation recommends comparing the template's output with the generated version before publishing.
A ready-to-copy template
The template first keeps Shopify's default groups using the official loop, then adds your groups as plain text, as the documentation describes for custom rules.
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /
# Search and on-demand retrieval: allowed,
# with the same exclusions as the * group
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
{%- for group in robots.default_groups -%}
{%- if group.user_agent.value == '*' -%}
{%- for rule in group.rules %}
{{ rule.directive }}: {{ rule.value }}
{%- endfor -%}
{%- endif -%}
{%- endfor %}
The second loop copies every rule from Shopify's * group (cart, checkout, account, collection filters, and so on) under the search crawlers. This is essential: a crawler that finds a group with its own name follows only that group, so it would ignore the exclusions in the * group. Because the rules are read from robots.default_groups, they follow Shopify's updates. The {%- and -%} markers strip extra spaces and line breaks so that each rule sits on its own line.
Remove from the first group any crawler you are willing to accept. Keep in mind that Google-Extended also covers grounding of answers in the Gemini apps.
Shopify-specific limits and pitfalls
- Don't replace everything with plain text. You can, but Shopify says it is "strongly recommended" to keep using the provided Liquid objects: a fixed file stops picking up the platform's updates to the default rules. The example above keeps the official loop.
- An old template can freeze outdated rules. The Help Center notes that an existing template that reproduces the default groups may keep rules the default file no longer has, such as
Disallow: /search. To start again from the current version, delete the template and recreate it. - The template belongs to the theme. It lives in the theme's
templatesfolder, so a different published theme won't necessarily contain it. Shopify also states that uploading a theme from the Themes section of the admin won't importrobots.txt.liquid, whereas Theme Kit and the command line preserve it. - Whitespace mistakes. A string added inside the loop without a line break can stick to the previous rule (
...remoteDisallow: ...), which makes both rules invalid. Always read through the rendered file. - Per-domain rules. The
request.hostobject allows rules specific to one domain, but Shopify reserves it for stores that use Shopify Markets with separate domains. - Duplicate names. If Shopify one day adds a default group for one of the crawlers you name, RFC 9309-compliant crawlers will merge the two groups. Check the result after any significant change.
To go back to the default file, open Edit code again, right-click robots.txt.liquid, select Delete permanently, then click Delete.
Check the result
Changes take effect on the store right away, but crawlers cache robots.txt (RFC 9309 asks them to refresh it at least every 24 hours), so allow a day. Open https://www.example.com/robots.txt and check three things: Shopify's default groups are still there, your two groups appear exactly once, and each rule under the search crawlers is on its own line. Then test specific URLs (a product page, /cart, /checkouts/) for GPTBot and OAI-SearchBot with the IABIG robots.txt tester: the product page should be disallowed for the first and allowed for the second, and the cart disallowed for both.
Official sources
- Shopify, Editing robots.txt.liquid: https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
- Shopify, Customize robots.txt: https://shopify.dev/docs/storefronts/themes/seo/robots-txt
- Shopify, Liquid
robotsobject: https://shopify.dev/docs/api/liquid/objects/robots - Shopify, Shopify agentic storefronts: https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts
- Shopify, Managing agentic storefronts: https://help.shopify.com/en/manual/online-sales-channels/agentic-storefronts/agentic-home
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP RestrictionsOn IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate LimitsA tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP RangesDeny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
- Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP FilterWhere to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.