Who actually crawls your store? 15 days of server logs

Plenty gets written about AI crawlers and very little gets measured. So I went through my own server log: 15 days, one server, one grep. Here's what came out.

So the numbers can be placed, here's the site they come from: markusstoeger.com, my own developer site. 164 URLs according to the sitemap, 83 German and 81 English, mostly technical guides on WordPress, WooCommerce and AI integration. No store, no ad budget, no ads. A mid-sized specialist site — nothing that attracts unusual attention.

| Crawler | Hits | | --- | --- | | meta-externalagent | 3,530 | | ClaudeBot | 1,131 | | GPTBot | 950 | | ChatGPT-User | 855 | | OAI-SearchBot | 623 | | AI crawlers total | 7,138 | | Googlebot | 2,581 | | bingbot | 982 |

The AI bots showed up almost three times as often as Google. And even taking out Meta's crawler, by far the greediest of them, it's still 3,559 against 2,581.

Spread across 164 pages that means every single URL was fetched by an AI crawler 43 times in 15 days on average. Just under three times a day, per page.

This isn't a forecast about the future of search. It's last week and the week before.

Three bots from OpenAI, three different jobs

Three OpenAI crawlers appear in that table, 2,428 hits between them. Lumping them together is the most common mistake I see — each does something different to your store.

GPTBot collects content for model training. What it takes ends up in the model itself and is why ChatGPT knows your brand even without looking anything up.

OAI-SearchBot builds the search index ChatGPT assembles its answers from. It's the reason you can appear as a source at all.

ChatGPT-User isn't a crawler in the classic sense. It runs when somebody is sitting in a chat right now, asking something your page might answer. At 855 hits it was the most active of the three for me — the homepage alone got fetched live 403 times.

That distinction is why blanket blocking is so expensive. Shut out everything that looks like AI and the same click hits the training bot and the two that actually bring visitors.

The most striking number isn't in the table

Of OAI-SearchBot's 623 hits, 447 went to a single file: robots.txt. Only the rest spread across actual content.

So it checks whether it's allowed before nearly every visit. That's good news and awkward news at once. Good, because a fix to robots.txt lands within hours rather than at the next big crawl. Awkward, because an accidental Disallow works just as fast — and nobody sends you a notification when it happens.

What belongs in that file I've written up in the guide to robots.txt for AI crawlers, template included.

And what comes out at the end?

12 clicks in the same period, with chatgpt.com as the referrer. Plus 6 from claude.ai.

That's not much, and I'm writing it down deliberately, because this is the point where big numbers usually get claimed. More interesting than the volume is which pages they were: technical articles with a clear question in the title, every one of them. No homepage, no services page. People arriving from ChatGPT arrive with a specific question and land on the page that answers it.

For a store that means: the product page made of manufacturer copy and a photo gallery stands little chance. The page that genuinely answers "does size 43 run true on this model?" stands some.

What of this transfers to your store

My 164 pages are a manageable specialist site. A WooCommerce store with 400 products, plus categories, filter pages and variants, quickly reaches a multiple of that — and gets fetched accordingly more often. At 43 hits per page over the same period that's tens of thousands of requests in two weeks, from AI crawlers alone.

That has two consequences which have nothing to do with visibility:

It costs server load. Crawlers don't ignore your page cache, but they reliably hit the corners nobody visits and which therefore never sit in cache. If your store is already running at its limit, you'll feel it.

They also fetch pages useful to nobody. Filter URLs with appended parameters, cart, checkout, customer account. The crawler burns time there and finds nothing quotable. That's why the robots.txt template has a Disallow for exactly those areas — not for secrecy, but so the available hits fall on your product and guide pages instead.

How to check this on your own site

You don't need server access for it.

In the browser. Open yourstore.com/robots.txt. That's the permission everything hangs on — and the file OAI-SearchBot fetched hundreds of times from me.

In analytics. Filter your referrers for chatgpt.com, perplexity.ai and claude.ai. In Google Analytics that's under Acquisition → Traffic acquisition, in Matomo and Plausible it's right there in referrers. That shows you clicks, not crawlers.

In WordPress. Several free plugins now log AI crawlers and list them in the dashboard. Search the plugin directory for "AI Crawler Analytics"; the decent ones keep the data in your own database and send nothing outside.

With server access it's quickest:

grep -c "OAI-SearchBot" /var/log/nginx/access.log
grep "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

The first line counts the hits, the second shows which pages it took. On Apache the log sits at /var/log/apache2/access.log; on managed hosting it's usually in the customer panel under "access statistics" or "raw logs".

Half an hour and you stop guessing. My result was well off what I expected — I'd have put Google in front.

---

Want the next build and audit guides, with all the traps? Subscribe to the "Built with AI" newsletter.

Rather done-for-you? If you want your store checked for AI visibility — crawler hits, robots.txt, firewall rules, structured data: drop me a line. It's my day job.