Is your store locking ChatGPT out? What really belongs in your robots.txt

I looked up which file on my own site OpenAI fetches most often. Not the homepage. Not a blog post. The robots.txt — 447 times in 15 days, from OAI-SearchBot alone. That is 72 percent of everything it requested.

The numbers come from the server log of markusstoeger.com, a technical site with 164 URLs. Nothing spectacular, no store, no ad budget. Yours probably looks similar — you just haven't looked yet.

So the bot asks permission again before nearly every visit. Which is exactly why this one text file decides whether your store shows up in ChatGPT answers at all.

The problem: on most WooCommerce stores, what's in there was never put there on purpose.

There are two kinds of AI crawler, and they get confused constantly

When the WordPress world talks about "blocking AI bots", it almost always means one kind. There are two, and they do completely different things.

Training bots collect content for model training. OpenAI's is called GPTBot, Anthropic's ClaudeBot, Meta's meta-externalagent. What they take ends up inside the model — it's what makes an AI know your brand exists.

Citation bots work for the search function. OAI-SearchBot builds the index ChatGPT assembles its answers from, and ChatGPT-User fetches a page live while someone is asking about it. These two put you in the answer — with a link, a source, a click.

The two kinds have nothing to do with each other. They come from different servers, carry different user agents and read robots.txt separately. A Disallow for GPTBot has zero effect on whether ChatGPT cites you.

But blanket-blocking anything that looks like AI hits both. The result is absurd: you protect product descriptions nobody wanted to steal, and disappear from the answers where buying decisions happen.

Who locks your store out without asking

In the stores I look at, the owner almost never blocked anything. Three others did:

Security plugins. Wordfence and Sucuri treat unknown user agents as a potential threat. A crawler their rule sets don't recognise gets thrown out, by the same logic that stops scrapers. For that purpose it's right. For OAI-SearchBot it's fatal.

Cloudflare. Since mid-2025 Cloudflare blocks AI crawlers by default on newly added domains and on the free plan. Anyone who put their store behind it without walking through the bot settings has been locking crawlers out since day one, without knowing.

Your SEO plugin. Yoast and Rank Math write their own robots.txt, overriding the file on disk. What you uploaded by FTP may not be what actually gets served.

The test takes two minutes: open yourstore.com/robots.txt in a browser. What you see there is what the crawlers see — regardless of what you entered somewhere else.

The template

Here's the file with both kinds handled properly:

# Citation bots: must be allowed, or you are invisible inside ChatGPT
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Training bots: your call
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

# Everyone else: content yes, internal endpoints no
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /my-account/

Sitemap: https://yourstore.com/sitemap.xml

Two notes on that.

Checkout, cart and account belong out. Not out of secrecy, but because those pages are worthless to a crawler: dynamic, personalised, uncacheable. It burns your crawl budget there and finds nothing it could quote.

On training bots you can reasonably disagree. A publisher who lives off their writing has good reasons to shut GPTBot out. A store doesn't. Your product descriptions aren't a protected work — they're sales material, and the more models know your brand, the more often its name comes up when someone asks for that kind of product. I let them all in.

What to check after the change

The robots.txt is only the permission. Whether it lands, you see elsewhere:

• In the browser: open yourstore.com/robots.txt and read what is actually served. A Disallow: / under one of the citation bots, or a blanket one for everybody — that's your cause.
• In the firewall: Wordfence shows blocked crawlers under Live Traffic. Cloudflare has the AI crawler settings in the dashboard under Security.
• In your traffic: a free WordPress plugin for AI crawler statistics shows you in the dashboard which bots actually come by. Search the plugin directory for "AI Crawler Analytics".

After a change it doesn't take a week. OAI-SearchBot reads the file several times a day — 447 times in two weeks, in my case. Let it in and it notices fast.

---

Want the next build and audit guides, with all the traps? Subscribe to the "Built with AI" newsletter.

Rather done-for-you? If you want me to look at your store — robots.txt, firewall rules, crawl budget and whatever else stands between you and AI visibility: drop me a line. It's my day job.