Open your raw web server access logs and search the user-agent field for the names AI companies publish, such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Perplexity-User, then count the matching requests per bot. Before you trust the numbers, verify the IP addresses, make sure each request is counted once, and separate bulk crawling from the live fetches an assistant makes while it is answering a real person.
Why don't AI crawlers show up in Google Analytics?
Analytics tools count a visit when a browser loads your page and runs the tracking script. Crawlers fetch your HTML to read it. You cannot count on them running your tag, and most analytics products filter known bots anyway. An empty "AI" segment in your analytics tells you nothing about crawlers.
The record that does exist is the server access log: one line per request, written by nginx, Apache, your load balancer or your CDN, whether the visitor is a person or a bot. On managed hosting, look for "raw access logs" in the control panel. If you sit behind a CDN, its logs may be the only place some bot traffic appears, because blocked or cached requests never reach your origin server.
In nginx's default combined format, each line holds the IP address, the time, the request line, the status code, the bytes sent, the referrer and the user agent. The last two quoted fields are the ones you need. For a list of which company runs which bot, see What Are GPTBot, OAI-SearchBot, ClaudeBot and Other AI Crawlers?.
Which commands find AI crawlers in server logs?
Split each line on double quotes. In the combined format, field 2 is the request, field 4 the referrer and field 6 the user agent. Matching only field 6 avoids false hits from referrers and URLs that happen to contain a bot's name.
Count requests per crawler across current and rotated logs:
zcat -f /var/log/nginx/access.log* \
| awk -F'"' 'match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Meta-ExternalAgent|Applebot|Amazonbot|Googlebot|bingbot/) {print substr($6, RSTART, RLENGTH)}' \
| sort | uniq -c | sort -rn
zcat -f reads plain and gzipped files alike. match takes the first bot name in the user agent and prints it once, so every log line counts as exactly one request.
See which pages a given bot fetched:
zcat -f /var/log/nginx/access.log* \
| awk -F'"' '$6 ~ /GPTBot/ {split($2, r, " "); print r[2]}' \
| sort | uniq -c | sort -rn | head -30
Check which status codes a bot receives:
zcat -f /var/log/nginx/access.log* \
| awk -F'"' '$6 ~ /ClaudeBot/ {split($3, s, " "); print s[1]}' \
| sort | uniq -c
Mostly 200s is what you want. A wall of 403 or 503 responses means something in front of your site, often a firewall rule or a CDN bot setting, is turning the bot away. How to Fix GPTBot Being Blocked by Cloudflare walks through the usual culprit.
Finally, see who reads your robots.txt:
zcat -f /var/log/nginx/access.log* \
| awk -F'"' '$2 ~ /robots.txt/ {print $6}' \
| sort | uniq -c | sort -rn | head
A bot that fetches robots.txt and then nothing else has usually been told to stay out.
Why do my crawler counts look too high?
This is the double-counting trap, and it inflates numbers in four common ways.
Counting tokens instead of requests. grep -o prints every match, not every line. Some crawlers write their name more than once in the user agent, for example as the product token and again in a contact address, so a broad case-insensitive pattern counts one request twice. A loose pattern such as Claude also lumps together ClaudeBot, Claude-User and Claude-SearchBot, which Anthropic describes as three separate agents with different jobs: training, fetching pages when a person asks Claude something, and search indexing.
Matching the whole line. A plain grep catches the bot name in the referrer or in a URL like a blog post about GPTBot, and it counts human readers of that post as bots.
Reading the same traffic twice. A decompressed copy sitting next to its .gz original, CDN logs added on top of origin logs, or two virtual hosts logging the same request will all double your totals.
Counting names that never appear. Per Google's documentation, Google-Extended is a robots.txt control token layered on existing Googlebot requests, not a separate crawler. It will never show up as its own entry in your logs. A report showing "Google-Extended visits" is counting something else.
When we report crawler volume, we count requests, not user-agent tokens.
How is crawling different from live retrieval?
These are different events, and only one of them means a person is using your page right now.
| Kind of visit | Agents you will see | What triggers it | What it tells you |
|---|---|---|---|
| Training crawl | GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot | The company's own crawl schedule | Your content may enter training data; says nothing about any specific answer |
| Search indexing | Claude-SearchBot, PerplexityBot, Googlebot, bingbot | Building the indexes assistants search | Your pages can be found when an assistant searches |
| Live retrieval | ChatGPT-User, OAI-SearchBot, Perplexity-User, Claude-User | A person asking a question at that moment | Your page was fetched for a real answer |
The scale gap is large. Across the full retained nginx logs of pickaclass.com, an edtech platform we work with, from 28 July to 11 August 2026, ClaudeBot made 19,272,125 requests. In the same fortnight the platform received 1,918 live retrieval fetches (ChatGPT-User, OAI-SearchBot, Perplexity-User) on content pages. A client blog of a few dozen articles received 56 live fetches across 24 different articles.
List your live fetches by page:
zcat -f /var/log/nginx/access.log* \
| awk -F'"' 'match($6, /ChatGPT-User|OAI-SearchBot|Perplexity-User|Claude-User/) {split($2, r, " "); print substr($6, RSTART, RLENGTH), r[2]}' \
| grep -vE '(robots.txt|sitemap|[.](css|js|png|jpg|svg|ico|woff2?))' \
| sort | uniq -c | sort -rn | head -50
Then check whether humans clicked through from an assistant, by referrer, excluding bots:
zcat -f /var/log/nginx/access.log* \
| awk -F'"' '$4 ~ /chatgpt.com|perplexity.ai|claude.ai|gemini.google.com|copilot.microsoft.com/ && $6 !~ /bot|Bot|-User/ {print $4}' \
| sort | uniq -c | sort -rn
Expect this to be small. In about two weeks of logs from three of our course websites, roughly 1.2 million requests in the two busiest logs, human visitors arriving from AI assistants totalled one from Copilot and zero from ChatGPT, Perplexity, Gemini, Claude, Meta AI and others. Assistants mostly answer in place rather than send a click, so live fetches are a better signal of use than referral traffic.
How do I know a request really came from that bot?
Anyone can put "GPTBot" in a user agent. Verify before you draw conclusions.
For Google, Google Search Central describes the method: run a reverse DNS lookup on the IP, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and check it resolves back to the same IP. On Linux, host on the IP followed by host on the returned name does both steps.
Perplexity publishes machine-readable IP ranges for PerplexityBot that you can cross-check against the IPs in your logs. Where a company publishes ranges, use them. A "ClaudeBot" request from a residential IP address is almost certainly a scraper borrowing the name, and it belongs outside your counts.
What did our own logs show?
Three things stood out.
AI crawlers dwarf Googlebot on a large site. On pickaclass.com, 28 July to 11 August 2026: ClaudeBot 19,272,125 requests, Meta-ExternalAgent 3,854,076, GPTBot 2,744,985, Applebot 2,655,555, Amazonbot 1,170,268, Googlebot 52,292. That site has about 190,000 crawlable URLs and is crawled constantly. A different site with complete markup (server rendering, Article, FAQPage and Organization JSON-LD, a sitemap) but only 75 URLs was still absent from answers on its own article's exact topic. Surface beat markup.
One robots.txt line can silence everything. On our own site, a rule meant to hide a single redirect page, /b, was written as a prefix. Because robots.txt rules match prefixes, from 11 August to 21 October 2026 it also closed /blog, /business and every article in our business blog to all crawlers. RFC 9309 says a crawler that has downloaded robots.txt must follow its parseable rules, and they did. Bing Webmaster Tools listed our business page as "discovered but not crawled"; Google had none of the business pages indexed. We changed the rule to /b$ on 21 October 2026. Logs make this kind of mistake visible: robots.txt requests arrive, page requests do not. Is My Website Blocking GPTBot, ClaudeBot and PerplexityBot? covers the checks.
Being crawled or indexed is not being found. On 19 October 2026 we asked assistants for the exact titles of 18 articles, each older than three weeks, on three client sites. ChatGPT found 1 of 18; Claude found 7 of 18. Google had indexed 29 of 30 articles of one of those sites.
What should you do with what you find?
If no AI bots appear at all, check robots.txt and your CDN or firewall first. If bots get 200s but you see no live fetches, assistants can read you but are not using you: you need pages that directly answer the questions your buyers ask, and more of them. If live fetches appear, note which pages they hit; those are the pages assistants already rely on.
Then measure the answers themselves, because logs show reading, not naming. Answers vary: the same question asked three times often names different businesses, so ask each question several times. How to Check Whether AI Assistants Mention Your Business explains the method, and the free check at ItGrows shows in about 30 seconds whether ChatGPT and Perplexity name you for five of your buyers' questions. Nobody can guarantee an assistant will name a business.
Frequently Asked Questions
Can I see AI crawlers in Google Analytics?
Not reliably. Analytics records visits when a browser runs the tracking script, and bots are usually filtered or never run it. Use raw server or CDN access logs instead.
Does a visit from GPTBot mean ChatGPT will recommend me?
No. GPTBot is a crawler, and crawling says nothing about a specific answer. In our 19 October 2026 test, ChatGPT found only 1 of 18 client articles by exact title, even though Google had indexed nearly all of one site's articles.
Why is ChatGPT-User in my logs but I get no visitors from ChatGPT?
ChatGPT-User fetches your page while answering someone, and the assistant usually answers in place without sending a click. In two weeks of logs from three course websites we saw one human visitor from Copilot and none from ChatGPT.
Should I block ClaudeBot if it sends millions of requests?
That is a choice about training access, and Anthropic lets you control ClaudeBot, Claude-User and Claude-SearchBot separately in robots.txt, including with Crawl-delay. Blocking ClaudeBot alone does not block the agents that fetch pages for live answers or search, so decide per agent rather than blocking everything with Claude in its name.