AI Crawlers Explained: Which Bots to Allow in robots.txt

When a buyer asks ChatGPT, Claude, Perplexity or Google's AI for the best tool in your category, the answer is built from pages those systems were able to read. If their crawlers can't reach your site, you can't be quoted, however good your content is. That makes crawler access the first, cheapest check in any AI search optimization plan.

This guide explains the main AI crawlers, what each one is for, how to control them in robots.txt, and the mistakes that quietly remove companies from AI answers.

Training crawlers vs search crawlers vs user fetches

The most useful thing to understand is that AI companies now run several bots, each with a different job, and you can treat them differently.

• Training crawlers collect public pages that may be used to train future models. Blocking them keeps your content out of future training data. It doesn't remove you from today's AI search answers.
• Search crawlers build the index an assistant searches when it answers a question with live sources. Blocking them means the assistant can't cite your pages in those answers.
• User-initiated fetches happen when a person asks an assistant to open a specific page. Some companies say robots.txt may not apply to these, because a person asked for the page directly.

Mixing these up is the most common error we see: a team decides to "block AI" to protect its content, blocks every AI user agent, and then wonders why ChatGPT search never mentions them.

The main AI crawlers and what they do

User agent names are what you write in robots.txt. Companies add and rename bots from time to time, so check each vendor's current documentation before you change rules on a live site.

OpenAI (ChatGPT)

• GPTBot: the training crawler. OpenAI says blocking it keeps your content out of training for its foundation models.
• OAI-SearchBot: the search crawler for ChatGPT search. OpenAI says sites that block it won't be shown in ChatGPT search answers, and that changes to robots.txt take about 24 hours to take effect.
• ChatGPT-User: fetches pages when a user's request needs them. OpenAI notes robots.txt rules may not apply to these user-initiated visits, and that it doesn't use this agent to decide what appears in search.

OpenAI treats each setting separately, so you can allow OAI-SearchBot while disallowing GPTBot.

Anthropic (Claude)

• ClaudeBot: collects public web content that may be used for training.
• Claude-SearchBot: indexes pages to improve the search results Claude uses to answer questions.
• Claude-User: fetches a page when a Claude user asks for it.

Each needs its own robots.txt rule: blocking ClaudeBot doesn't block Claude-SearchBot or Claude-User.

Perplexity

• PerplexityBot: indexes pages so Perplexity can show and link them in answers.
• Perplexity-User: fetches pages when a user's question needs them.

Google (Gemini and AI Overviews)

Google doesn't use a separate crawler for its AI features. Googlebot crawls your site for Search, and AI Overviews and AI Mode are part of Search, so blocking Googlebot removes you from both. Google-Extended is a robots.txt token, not a crawler: it controls whether content Googlebot collects can be used to train and ground Gemini models outside Search. Blocking Google-Extended doesn't remove you from Google Search or AI Overviews. For more on AI Overviews, see why AI Overviews cut your traffic.

Microsoft, Apple and others

• Bingbot crawls for Bing, which also feeds Microsoft Copilot answers. Blocking Bingbot removes you from both.
• Applebot-Extended is a token, like Google-Extended, that controls whether Applebot's data can be used to train Apple's AI models.
• CCBot is Common Crawl's crawler. Its public dataset is widely used to train AI models.
• Meta-ExternalAgent, Amazonbot and Bytespider are other crawlers you may see in your logs from Meta, Amazon and ByteDance.

Should a SaaS company block AI crawlers?

For most SaaS companies, no, at least not the search crawlers. Your marketing site exists to be found and understood. Buyers increasingly ask AI assistants for shortlists, and blocking the bots that build those answers removes you from consideration. The training crawlers are a real choice with trade-offs.

• Reasons to allow training crawlers: future models learn what your company does and how to describe it, which can shape answers even when no live search happens.
• Reasons to block them: you publish original research, paid content or proprietary data you don't want reused, or your legal team prefers to opt out.
• What we recommend for a typical SaaS marketing site: allow everything public, protect what is genuinely private behind a login (robots.txt is a request, not security), and if you do block training crawlers, keep the search crawlers allowed.

A robots.txt you can copy

This is the shape of the robots.txt we recommend for a SaaS marketing site that wants to be cited in AI answers. Write it as plain text at the root of your domain (yourdomain.com/robots.txt).

• User-agent: * then Allow: /, plus Disallow: lines for private areas such as /admin and /api.
• A separate block allowing each search crawler explicitly: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot and Googlebot.
• Your choice for training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Allow them, or add Disallow: / under each one to opt out.
• A Sitemap: line pointing to your XML sitemap, so every crawler finds all your pages.

Explicit rules aren't required if your wildcard rule already allows everything, but they make your intent clear and protect you if someone later adds a broad block. Our own site allows every major AI crawler and blocks only admin and API paths.

The mistakes that hide sites from AI answers

1. A firewall or CDN bot rule blocks them. Many hosting and CDN providers now offer one-click AI bot blocking, and some turn it on by default. Your robots.txt can say "allow" while your firewall returns an error. Check your CDN's bot settings.
2. Blocking everything to stop training. Teams add a blanket block for every AI user agent and lose search visibility too. Separate training from search.
3. Content that only appears with JavaScript. Some AI crawlers don't run JavaScript, or run very little. If your key copy, pricing or FAQs only load in the browser, they may not see it. Server-render important pages.
4. Rate limits that are too tight. Aggressive rate limiting can return errors to crawlers during a busy fetch. Watch for spikes of 429 or 403 responses in your logs.
5. Forgetting staging rules. A Disallow: / copied from a staging site to production is still one of the most common technical SEO accidents.
6. Stale user agent names. Rules written for old bot names do nothing for the current ones. Review the list every few months.

How to check whether AI crawlers can reach your site

1. Read your robots.txt at yourdomain.com/robots.txt and look for broad Disallow: / rules under any AI user agent.
2. Check your server or CDN logs for the user agents above. If you never see OAI-SearchBot, Claude-SearchBot or PerplexityBot, something may be blocking them, or they haven't discovered you yet.
3. Check your firewall and bot management settings for AI bot blocking.
4. Fetch your key pages without JavaScript (for example, view the page source) and make sure the main copy is in the HTML.
5. Test in the assistants themselves. Ask ChatGPT, Claude and Perplexity about your company and category, and see whether they cite your pages. Our guide to tracking your brand in AI search shows how to do this every month.

What about llms.txt?

llms.txt is a proposed file that gives AI tools a short, plain-text summary of your site and links to your most useful pages. It doesn't control access the way robots.txt does, and support for it varies between AI companies. It's cheap to add and harmless, but it won't fix a blocked crawler. Read the llms.txt glossary entry for the details.

Access is step one, not the whole job

Letting crawlers in makes you eligible to be cited. Whether you are cited depends on what they find: clear, specific answers, consistent descriptions across the web, structured data, reviews and third-party mentions. That's the work of answer engine optimization, and our GEO vs AEO vs SEO guide explains how it fits with classic SEO.

Frequently asked questions