AI crawler user agents are the names that OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others use when their software requests your pages. Each one does one of three jobs: collecting content for model training, building a search index the assistant cites from, or fetching a page live while a person is waiting for an answer. You control most of them separately in robots.txt.
This is a reference page. It covers who runs each agent, what it is for, how to allow or block it, and the mistakes we have seen, including one on our own site. Vendors rename and add agents from time to time. Before you change a production robots.txt, check the crawler documentation that each vendor publishes on its own site.
Which AI crawlers exist and who runs them?
The table lists the agents you are most likely to see in your logs. The middle column is the one that matters for visibility. Blocking a training crawler and blocking a search or retrieval agent have very different consequences.
| User agent | Operator | Job | What blocking it does |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models | Keeps your pages out of future training data. Does not by itself remove you from ChatGPT search |
| OAI-SearchBot | OpenAI | Crawls pages so they can be shown and linked in ChatGPT search answers | Your pages can stop appearing as cited results in ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user or a custom GPT asks for it | Stops live fetches. OpenAI has said robots.txt may not apply in the same way to user-initiated requests |
| ClaudeBot | Anthropic | Collects content for training Claude models | Keeps your pages out of Anthropic's training data |
| Claude-SearchBot | Anthropic | Crawls to improve Claude's search results | Can reduce how often Claude's search finds your pages |
| Claude-User | Anthropic | Fetches a page when a Claude user's question needs it | Claude cannot read your page while answering |
| PerplexityBot | Perplexity | Builds the index Perplexity uses to surface and cite sites. Perplexity says it is not used to train foundation models | Your pages can drop out of Perplexity answers |
| Perplexity-User | Perplexity | Fetches a page in response to a user's question | Perplexity has said these user-requested fetches generally do not follow robots.txt |
| Google-Extended | A robots.txt token, not a separate crawler. It controls whether content Googlebot already crawls is used for Gemini training and grounding | Opts out of that use. Google says it does not affect Google Search, and AI Overviews are part of Search | |
| Googlebot | Google Search crawler, which also feeds AI Overviews | Removes you from Google Search, and with it AI Overviews | |
| Bingbot | Microsoft | Bing search index, which Copilot draws on | Removes you from Bing and the assistants that rely on it |
| Applebot / Applebot-Extended | Apple | Applebot crawls for Siri and Spotlight. Applebot-Extended is a token that controls use for training Apple's models | Blocking Applebot-Extended opts out of training. Applebot still crawls |
| Meta-ExternalAgent / Meta-ExternalFetcher | Meta | The first crawls for AI training and related uses. The second fetches links on a user's request | Opts out of Meta's AI crawling or user-triggered fetches |
| Amazonbot | Amazon | Crawls for Amazon services, including Alexa answers | Your content is not available to those services |
| CCBot | Common Crawl | Builds an open web archive that many AI developers use as training data | Keeps future crawls of your site out of that archive |
Two things in the table catch people out. Google-Extended and Applebot-Extended never appear in your logs, because they are permission tokens read by the main crawler. And Gemini answers that come through Google Search rely on Googlebot, not on a separate AI bot. Our article on showing up in Gemini answers and AI Overviews covers that side.
What is the difference between training, search and live retrieval agents?
Think of the three jobs as three different moments.
Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect pages that may end up in a future model. That model ships months later and cannot cite the page it learned from. Blocking them is a decision about whether your content trains someone's model. It is not a decision about being cited next week.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) build the indexes an assistant searches when it decides to look something up. If you want an assistant to cite and link your page, these are the agents that need access.
Live retrieval agents (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) fetch one page at the moment a real person asks a question. In logs we have reviewed, they are the clearest sign that an assistant is using your content in an actual answer. On the edtech platform pickaclass.com, between 28 July and 11 August 2026, we counted 1,918 live fetches by ChatGPT-User, OAI-SearchBot and Perplexity-User on content pages. A client blog of a few dozen articles received 56 across 24 different articles in the same fortnight.
An assistant does not search for every question. In a test on 23 September 2026 of 20 German business questions, the model answered 10 without searching and cited nobody. Training data covers those answers. Search and retrieval access covers the rest.
How do you allow or block each agent in robots.txt?
Each agent reads the robots.txt group that names it. If a group matches its name, it follows that group and ignores the User-agent: * group. That lets you treat training and search differently.
Here is a common setup for a business that does not want its content used for training but does want to be cited:
```
Training: opt out
User-agent: GPTBot Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: Google-Extended Disallow: /
User-agent: Applebot-Extended Disallow: /
Search and live retrieval: allow
User-agent: OAI-SearchBot Allow: /
User-agent: ChatGPT-User Allow: /
User-agent: Claude-SearchBot Allow: /
User-agent: Claude-User Allow: /
User-agent: PerplexityBot Allow: /
User-agent: * Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml ```
If you want maximum visibility, the simplest robots.txt has no AI-specific blocks at all, just a sensible * group and a sitemap line. To block everything AI-related, give each agent its own Disallow: / group. A * rule will not catch an agent that has its own named group elsewhere in the file.
robots.txt is a request, not a lock. The major vendors say their crawlers honour it, with the caveats about user-initiated fetches noted in the table. If you need a hard block, do it at the server or CDN by user agent and verify against the IP ranges vendors publish. User agent strings are easy to fake.
What goes wrong when sites try to control AI crawlers?
Prefix rules that block more than intended. robots.txt rules match prefixes. On itgrows.ai, a rule meant to hide a single redirect page, /b, was written as Disallow: /b. From 11 August to 21 October 2026 it also closed /blog, /business and every article in our business blog to all crawlers. Bing Webmaster Tools listed our business page as "discovered but not crawled" from 11 August 2026, and Google had none of the business pages indexed. We changed the rule to /b$ on 21 October 2026. Read every Disallow line as "everything that starts with this".
The CDN or firewall blocks what robots.txt allows. Bot protection at the network edge can return errors to AI crawlers while your robots.txt says "allow". If your logs show no AI agents at all, check that layer. We wrote up the common GPTBot block in Cloudflare and how to fix it, and a broader check of whether your site is blocking GPTBot, ClaudeBot and PerplexityBot.
Blocking the search crawler to stop training. Disallowing every OpenAI or Perplexity agent because you object to training also takes you out of their search results. Block the training agent by name and leave the search agent alone.
Assuming Google-Extended controls AI Overviews. It does not. AI Overviews come from Google Search, so Googlebot access decides them.
Treating access as visibility. Allowing every crawler is necessary but not sufficient. On 19 October 2026 we asked assistants for the exact titles of 18 articles on three client sites, each older than three weeks. ChatGPT found 1 of 18 and Claude found 7 of 18, even though Google had indexed 29 of 30 articles of one of those sites. Search indexes matter too. ChatGPT's relationship with Bing is covered in Does ChatGPT use Bing?.
How much do these crawlers actually visit?
More than most site owners expect. Across the full retained nginx logs of pickaclass.com, from 28 July to 11 August 2026, we counted these requests:
- ClaudeBot: 19,272,125
- Meta-ExternalAgent: 3,854,076
- GPTBot: 2,744,985
- Applebot: 2,655,555
- Amazonbot: 1,170,268
- Googlebot: 52,292
That site has a very large crawlable surface, and crawl volume follows surface. A site with about 190,000 crawlable URLs was crawled constantly. A site with complete markup but only 75 URLs was absent from answers on its own article's exact topic.
Heavy crawling did not mean clicks. In about two weeks of logs from three course websites, we saw one human visitor arriving from Copilot and none from ChatGPT, Perplexity, Gemini, Claude or Meta AI. Assistants mostly answer in place. That is why live retrieval fetches are a more useful signal than referral traffic.
Which agents should a business that wants to be named allow?
If being named by assistants matters more to you than keeping content out of training sets, allow all of them. Training data shapes the answers given without a search, which in our September test was half of them. Search and retrieval access shapes the rest.
If you do want to opt out of training, block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent by name. Keep OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot open.
Then check whether any of it changes the answers. Asked the same question three times, an assistant often names different businesses. One test proves little. To see where you stand today, the free check at ItGrows takes about 30 seconds and shows whether ChatGPT and Perplexity name your business for five of your buyers' questions. Nobody can guarantee an assistant will name you, but you can see whether the door is open.
Frequently Asked Questions
Does blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot collects training data, while OAI-SearchBot crawls for ChatGPT search results, so they are controlled separately. Blocking GPTBot only keeps your pages out of future training.
Is there a separate crawler for Gemini?
No. Google-Extended is a robots.txt token that controls whether content crawled by Googlebot is used for Gemini training and grounding. AI Overviews belong to Google Search and depend on Googlebot access.
Do ChatGPT-User and Perplexity-User follow robots.txt?
Both vendors describe these agents as acting on a user's request. OpenAI has said robots.txt may not apply in the same way, and Perplexity has said Perplexity-User generally ignores it. If you need to stop them, block at the server or CDN level.
How do I see which AI crawlers visit my site?
Search your server or CDN access logs for the user agent names in the table above and count requests per agent. Verify suspicious traffic against the IP ranges the vendors publish, since user agent strings can be faked.