AI Search Handbook
Buy PDF $29
Technical GEO10 min read

AI crawler robots.txt guide: which bots to allow and how

Understand search, user-triggered and training crawlers; configure robots.txt for AI visibility; and test the CDN, firewall and rendered response.

By · Adapted from The AI Search Handbook

Separate discovery from training

Crawler names are easy to treat as one list, but they represent different choices. Search crawlers discover material for current search experiences. User-triggered agents fetch a page because a person asked. Training crawlers collect material that may be used to improve future models. A publisher can make a different policy decision for each purpose.

CrawlerTypical roleWhy it matters
GooglebotGoogle Search crawlingEligibility for Google Search, AI Overviews and AI Mode supporting links
bingbotBing search crawlingBing index and Microsoft search/Copilot discovery
OAI-SearchBotOpenAI search discoveryOpenAI says it should not be blocked for content to appear in ChatGPT search summaries and snippets
ChatGPT-UserUser-triggered page fetchLets ChatGPT retrieve a page in response to a user's request
GPTBotPotential model trainingA separate policy choice from ChatGPT search inclusion
ClaudeBot / anthropic-aiAnthropic crawlingAccess policy for Claude-related crawling
PerplexityBot / Perplexity-UserPerplexity crawling and user fetchesAccess policy for Perplexity discovery and direct retrieval
Google-ExtendedGoogle model training/grounding controlNot the control for Google Search indexing; Googlebot remains the search crawler

A permissive robots.txt pattern

User-agent: *
Allow: /
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The wildcard rule already permits named bots unless another matching group overrides it, but explicit groups make policy visible to reviewers. Keep private endpoints out of the crawl surface and never use robots.txt as access control for confidential data.

Robots permission is only the first gate

  • CDN or web application firewall: verified crawlers may be caught by generic bot rules.
  • JavaScript or CAPTCHA challenges: a crawler can be allowed by robots.txt and still receive a challenge page.
  • Authentication and regional rules: public search pages should not require a session or be blocked in crawler regions.
  • Rate limiting: repeated 429 responses can make an otherwise open site effectively unavailable.
  • Rendering: the server response must contain the important text, links and metadata without requiring a click.
  • Status and redirects: canonical URLs should return 200, while alternate hosts and paths should resolve through clean permanent redirects.

How to test an AI crawler path

  1. Fetch robots.txt and confirm the exact user-agent is not disallowed.
  2. Request the canonical page with the crawler's published user-agent and record the HTTP status, final URL and response body.
  3. Check the CDN and firewall logs for 403, 429, challenge or timeout responses.
  4. Confirm the crawler receives the same core claims, headings, canonical URL and structured data a visitor receives.
  5. Verify the page is present in the appropriate webmaster index and sitemap.
  6. Repeat after infrastructure, privacy or bot-protection changes.

OpenAI-specific checks

OpenAI's publisher guidance distinguishes OAI-SearchBot from GPTBot. OAI-SearchBot supports search discovery; GPTBot relates to potential training. OpenAI also publishes IP ranges for its search crawler. If robots.txt is correct but requests still fail, inspect hosting and security layers rather than adding more metadata.

Common mistakes

  • Allowing a crawler in robots.txt while a CDN silently blocks its requests.
  • Blocking all unknown bots and assuming verified AI crawlers will be exempt automatically.
  • Serving an empty application shell to crawlers while the useful copy appears only after client-side requests.
  • Listing a sitemap on the wrong host or filling it with redirected and non-canonical URLs.
  • Confusing a training opt-out with a search-indexing control.
  • Publishing private, personal or licensed material publicly because a crawler was allowed.

Primary sources and further reading

Book-derived explanations on this page are editorial synthesis. Platform-specific claims link to the platform's own current documentation.

Frequently asked questions

Should I allow GPTBot to appear in ChatGPT search?

ChatGPT search discovery is associated with OAI-SearchBot. GPTBot is a separate control related to potential model training.

Does Allow: / guarantee an AI crawler can access the site?

No. A CDN, firewall, CAPTCHA, authentication rule, rate limit or rendering failure can still block or degrade the response.

Is robots.txt a security control?

No. Robots.txt is a voluntary crawler directive. Confidential material must be protected with real authentication and access controls.