The three jobs, bot by bot
| Company | Training | Search index | Live user fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None (says its bots do not collect training data) | PerplexityBot | Perplexity-User |
| Meta | Meta-ExternalAgent | Meta-WebIndexer | Meta-ExternalFetcher |
| Google-Extended (robots.txt token only) | Googlebot | — | |
| Apple | Applebot-Extended (robots.txt token only) | Applebot | — |
| Common Crawl | CCBot | — | — |
A few rows need a note. Google-Extended and Applebot-Extended never appear in your logs, because they are not crawlers. They are names you use in robots.txt to say whether content that Googlebot or Applebot already fetched may be used for AI models. Google states that Google-Extended has no effect on inclusion or ranking in Google Search, and AI Overviews and AI Mode are built on the Search index, so Googlebot is the bot that matters for them.
Microsoft Copilot answers from the Bing index and has no user agent of its own; Vercel's 2024 crawler study left it out for that reason. Bingbot is the bot to watch for Copilot. OpenAI also runs OAI-AdsBot, which only visits landing pages submitted as ChatGPT ads.
What each visit tells you
Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) visit in bulk and irregularly. Their visits say nothing about whether you appear in answers today. Blocking them is a policy decision about model training. It does not remove you from AI search, because search uses a different bot.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Googlebot, Bingbot) decide whether your pages are in the index an assistant searches. If they cannot reach a page, the assistant cannot cite it. For AI search visibility these are the bots to watch. A new guide that no search crawler has fetched after a few weeks is worth investigating.
User fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) fire when a person's question leads the assistant to open your page there and then. A run of ChatGPT-User requests for one URL is a strong hint that the page is being used in answers, even when GA4 shows no referral, because the person may never click through.
Reading your logs
Group bot requests by user agent and by path, then look at three things.
- Coverage. Which of your important pages has each search crawler fetched in the last 30 days? Gaps point to crawl problems: slow responses, redirect chains, blocked resources or pages missing from the sitemap.
- Status codes. A bot receiving 404, 403 or 5xx responses is wasting its visits. A 403 for a bot you meant to allow usually means a firewall or CDN bot rule is overriding robots.txt.
- User-fetch hot spots. The paths with the most ChatGPT-User, Claude-User and Perplexity-User requests are the pages assistants are actually reading on someone's behalf. Compare them with your AI referral report in GA4 to see how many of those readings turn into visits.
This site's own admin console keeps the same view: every AI and search crawler by name, with the pages each one fetched and when it last came.
Make sure it is really the bot
User agent strings are easy to fake, and scrapers often borrow the names of well-known bots. Before acting on a pattern, verify it:
- Googlebot and Bingbot: do a reverse DNS lookup on the IP, check that the hostname ends in googlebot.com, google.com or search.msn.com, then do a forward lookup to confirm it resolves back to the same IP.
- OpenAI: publishes a separate JSON list of IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User.
- Perplexity: publishes IP lists for PerplexityBot and Perplexity-User.
- Anthropic: publishes the IP addresses its crawlers use, though it recommends robots.txt rather than IP blocking for opting out.
A request claiming to be GPTBot from a residential IP is not GPTBot. Treat it as an ordinary scraper.
Do user fetchers obey robots.txt?
Not always, and the providers say so. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts each action. Perplexity says Perplexity-User generally ignores robots.txt, and Meta says Meta-ExternalFetcher may bypass it. Anthropic says its bots, Claude-User included, follow robots.txt. If a page must stay away from assistants acting for users, put it behind authentication rather than relying on robots.txt.
Acting on what you find
Because the jobs are separate, robots.txt can treat them separately: a publisher can block the training crawlers and still allow the search crawlers, so it stays citable without contributing to model training. This site allows everything, because the books are free and wide reuse is the point. Either choice is reasonable as long as it is deliberate and the search crawlers are never blocked by accident. The AI crawler robots.txt guide has the rules and the CDN and firewall checks.
Primary sources and further reading
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Meta: Meta web crawlers
- Google Search Central: Google's common crawlers and Google-Extended
- Vercel: The rise of the AI crawler
Book-derived explanations on this page are editorial synthesis. Platform-specific claims link to the platform's own current documentation.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. ChatGPT search relies on OAI-SearchBot. Blocking GPTBot only affects training.
Why don't I see any AI bots in GA4?
Bots do not run your analytics tag, so GA4 never sees them. You need server logs or a server-side counter.
Is Google-Extended a crawler?
No. It is a robots.txt token with no user agent of its own. It controls whether content Google has crawled may be used for Gemini models, and it does not affect Google Search.