AI search and AEO
AI crawlers
Also called AI bots
AI crawlers are the bots AI companies use to read websites, either to train their models or to fetch pages live when answering a question. Examples include GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot.
Training bots vs search bots
Some crawlers collect pages to train future models. Others fetch pages in real time when a user asks a question, so the assistant can cite current sources. Blocking the second kind means an AI tool can't quote your site in its answers.
What to check
- Your robots.txt allows the AI search bots you want citing you.
- Your firewall or CDN isn't blocking them by mistake.
- Important content is in the HTML, not only loaded by JavaScript.
- Your server logs show the bots visiting your key pages.
Frequently asked questions
Most shouldn't block the search and user bots, because that removes them from AI answers. Blocking training bots only is a separate, policy decision.
Check your server or CDN logs for their user agents, such as GPTBot, ClaudeBot or PerplexityBot.