Technical SEO
robots.txt
robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may and may not access. It controls crawling, not indexing.
What it's for
- Keeping crawlers out of admin areas, internal search and duplicate parameter URLs.
- Setting rules for specific bots, such as AI crawlers.
- Pointing crawlers to your XML sitemap.
A common mistake
Blocking a page in robots.txt doesn't remove it from search: Google can still index the URL if other pages link to it. To keep a page out of results, allow crawling and use a noindex tag instead.
Frequently asked questions
At yourdomain.com/robots.txt. Every site should have one, even if it allows everything.
Yes, by naming their user agents, such as GPTBot. Most SaaS sites allow the AI search bots so they can be cited.