What is robots.txt?
Definition
robots.txt is a plain-text file at a site's root that tells search engine and AI crawlers which parts of the site they may and may not crawl.
Crawlers check robots.txt before fetching pages, and well-behaved ones follow it. Each group of rules names a user agent, such as Googlebot, Bingbot, or OpenAI's OAI-SearchBot, and lists the paths it may or may not crawl. The file can also point crawlers to the sitemap. Mistakes are common after site launches, when a staging rule that blocks everything is left in place.
Two limits are worth knowing. robots.txt controls crawling, not indexing: Google notes a blocked URL can still be indexed if other pages link to it. And blocking a crawler has consequences you may not intend. OpenAI, for example, says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Many sites now allow search crawlers while deciding separately about crawlers that collect data for AI training.
Why does it matter to your business?
One wrong line can hide your whole site from a search engine or an AI answer engine. Check which bots your file allows before assuming you are visible.
Related terms
Sources, verified September 11, 2026
Want to know how your site measures up?
The free audit checks whether Google and AI answer engines can find, understand, and quote your pages.