AI crawlers and AI search find your pages the same way search engines always have: by following links, reading your sitemap and robots.txt, and drawing on search indexes such as Bing and Google that assistants query while they answer. In our data, how many crawlable, linked pages a site had mattered far more than how perfect its markup was: a fully marked-up site with 75 URLs stayed out of answers on its own topic, while a site with about 190,000 crawlable URLs was crawled constantly.
The rest of this article explains the two routes into an assistant's answer, what sitemaps, links, IndexNow and the two big search indexes each contribute, and the mistakes that quietly lock you out.
What are the two ways an AI system reaches your pages?
There are two different kinds of visit, and people often blur them.
The first is bulk crawling. Bots such as ClaudeBot, GPTBot, Meta-ExternalAgent, Applebot and Amazonbot fetch pages in large volumes, ahead of any question. The volume is real. Across the full retained nginx logs of an edtech platform we work with (pickaclass.com), from 28 July to 11 August 2026, we counted ClaudeBot at 19,272,125 requests, Meta-ExternalAgent at 3,854,076, GPTBot at 2,744,985, Applebot at 2,655,555, Amazonbot at 1,170,268 and Googlebot at 52,292. Those are requests, not user-agent mentions.
The second is live retrieval. When a person asks ChatGPT or Perplexity something and the assistant decides to search, an agent such as ChatGPT-User, OAI-SearchBot or Perplexity-User fetches a page right then to build the answer. In the same fortnight the edtech platform received 1,918 such fetches on content pages; a client blog of a few dozen articles received 56, spread across 24 different articles.
The vendors treat these routes separately. OpenAI states that each of its crawler settings is independent, so you can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot from using your content for model training. Perplexity says PerplexityBot exists to surface and link websites in its search results and is not used to train foundation models. If you want to identify each bot by name, see what GPTBot, OAI-SearchBot, ClaudeBot and the others are and how to find them in your server logs.
One caveat: an assistant does not always search. In a 23 September 2026 test of 20 German business questions, the model answered 10 without searching at all and cited nobody. Where it did search, it cited primary sources such as laws, ministries, the OECD and large accounting firms on 9 of 10. Being findable is the entry ticket, not the prize.
How do sitemaps and internal links help crawlers discover pages?
A crawler can only fetch a URL it knows about. It learns URLs in three ways: links from pages it already has, your XML sitemap, and URLs it picks up from search indexes and other sites.
Internal links are the oldest and still the most dependable. Every new article should be linked from somewhere a crawler already visits: the blog index, a category page, related articles. An orphan page that exists only in a sitemap is a weak signal; a page linked from ten other pages tells a crawler it matters.
Sitemaps list every URL you want fetched, ideally with a last-modified date. They help most when a site is large or changes often. A sitemap does not force anything to be crawled or indexed. It is an invitation. Keep it clean: only canonical, live, indexable URLs, and reference it from robots.txt so any bot reading that file finds it.
External links matter because many crawlers and search indexes start from pages they already trust. A mention on a directory, a partner site or an industry publication can be how a bot first reaches you.
Do Bing, Google and IndexNow matter for AI search?
Yes, because live retrieval usually runs through a search index, not through the assistant's own fresh crawl of the web.
Google. Google's AI features are served by the same crawler as ordinary search. Google offers a separate robots.txt token, Google-Extended, for Gemini training and grounding, but that token has no effect on AI Overviews and AI Mode, which run on Googlebot. Google's guidance also says there is no special AI Overviews markup and no separate AI index. Getting indexed in Google Search is the route; Search Console tells you whether it worked.
Bing. Several assistants rely on web search to ground answers, and Bing's index is one of the main ones outside Google. Bing Webmaster Tools shows which of your URLs are discovered, crawled and indexed. We cover the details in does ChatGPT use Bing, and do you need Bing Webmaster Tools and IndexNow.
IndexNow is a simple protocol: when you publish or change a URL, your site pings participating search engines with that URL instead of waiting for them to notice. It shortens the gap between publishing and discovery. It does not guarantee indexing.
Here is the uncomfortable part. On 19 October 2026 we asked assistants for the exact titles of 18 articles, each older than three weeks, published on three client sites. ChatGPT found 1 of 18; Claude found 7 of 18. Google had indexed 29 of 30 articles on one of those sites. Being in Google's index did not mean an assistant could find the page. Each index and each assistant has to be checked on its own.
| Channel | What it does | What it does not do |
|---|---|---|
| Internal links | Lets any crawler reach a page from pages it already visits | Help a page nobody links to |
| XML sitemap | Lists every URL you want fetched, with change dates | Force crawling or indexing |
| IndexNow | Notifies participating engines of new or changed URLs immediately | Guarantee the URL gets indexed |
| Google indexing | Feeds Search, AI Overviews and AI Mode via Googlebot | Make a page findable in ChatGPT or Claude |
| Bing indexing | Supplies a major index that assistants can search while answering | Replace links, sitemaps or Google |
Why did surface area matter more than markup in our data?
We expected structured data to carry more weight than it did.
One site we looked at had everything a technical checklist asks for: server rendering, Article, FAQPage and Organization JSON-LD, and a sitemap. It had 75 URLs. It was still absent from answers on its own article's exact topic. Another site, with about 190,000 crawlable URLs, was crawled constantly.
The mechanism is not mysterious. Markup helps a crawler understand a page it has already fetched. It does nothing to get the page fetched, and it does nothing to put the page into the result set an assistant pulls when it searches. Surface area does both: more pages answering more specific questions means more URLs for crawlers to find, more entry points from links, and more chances to match the exact wording of a buyer's question.
So markup is worth having, but treat it as finishing, not foundation. If you have 20 pages, the gap between you and a competitor with 400 pages that each answer one buyer question will not be closed by JSON-LD.
What stops AI crawlers from finding your pages?
Most failures are self-inflicted and silent.
robots.txt rules that match more than you meant. RFC 9309, the standard for robots.txt, defines how rules control crawler access, and rules match by prefix. We learned this on our own site. On itgrows.ai, a rule meant to hide a single redirect page, /b, was written as a prefix. From 11 August to 21 October 2026 it also closed /blog, /business and every article in our business blog to all crawlers. Bing Webmaster Tools listed our business page as "discovered but not crawled" from 11 August 2026, and Google had none of the business pages indexed. We changed the rule to /b$ on 21 October 2026. On the same day, putting 20 English and 20 German questions to four assistants three times each, we were named in 0 of 160 answers and none of our pages was cited.
Blocking the wrong bot. Anthropic warns that blocking Claude-SearchBot may reduce a site's visibility in Claude's search results. Blocking a training crawler and blocking a search crawler are different decisions. Our guide on allowing GPTBot and other AI crawlers in robots.txt walks through the lines.
Firewalls and bot protection that return challenges or errors to bots, even when robots.txt allows them.
Slow propagation. Perplexity notes that robots.txt changes can take up to 24 hours to take effect. Fixing a rule today does not mean a visit today.
Client-side rendering that leaves crawlers with an empty page shell.
How do you check whether your pages are being found?
Work from the ground up.
- Read your robots.txt line by line. Test each Disallow against your real URL paths, not the one you had in mind.
- Check your logs for both bulk crawlers and live retrieval agents. Live fetches on specific articles are the clearest sign that an assistant is using a page in answers.
- Open Bing Webmaster Tools and Google Search Console. Look for "discovered but not crawled" and excluded pages, submit your sitemap, and set up IndexNow.
- Ask the assistants directly. Search for your exact article titles and for the questions your buyers ask. Ask each question more than once: answers vary from run to run, so a single ask tells you little.
- Don't count on clicks as proof. In about two weeks of logs from three of our course websites, roughly 1.2 million requests in the two busiest logs, we saw one human visitor arriving from Copilot and none from ChatGPT, Perplexity, Gemini, Claude or Meta AI. Assistants mostly answer in place, so your analytics will understate how often you are used.
For a quick start, the free check at itgrows.ai generates five buyer questions from your homepage and shows in about 30 seconds whether ChatGPT and Perplexity name you. ItGrows also runs this weekly for clients, asking ChatGPT, Claude, Gemini and Perplexity each buyer question three times and counting the answers that name the client and the sites cited. Nobody can guarantee an assistant will name a business, and we say so.
Frequently Asked Questions
Do AI crawlers use my XML sitemap?
A sitemap is one of the main ways any crawler learns which URLs exist, and referencing it in robots.txt makes it easy to find. It is an invitation rather than a command, so pair it with internal links to every page you care about.
If Google has indexed my page, will ChatGPT find it?
Not necessarily. On 19 October 2026 one client site had 29 of 30 articles in Google's index, yet ChatGPT found 1 of 18 articles across three sites by exact title. Check Bing and each assistant separately.
Does schema markup help AI search find my pages?
Markup helps a crawler interpret a page it has already fetched, but it does not get the page discovered. Google says there is no special AI Overviews markup, and in our data a fully marked-up 75-URL site stayed out of answers while a large site was crawled constantly.
Should I block GPTBot but allow OAI-SearchBot?
You can: OpenAI treats the two settings independently, so blocking training does not have to block ChatGPT search. Just be careful with the search bots, since blocking them removes you from the results those assistants show.