What is robots.txt?

Robots.txt is a text file in the root folder of your website that tells search engines which parts they may and may not crawl. It steers crawler behaviour and helps save crawl budget on unimportant pages. It is a basic tool in the technical SEO of every website.

What does robots.txt do?

With rules such as “Allow” and “Disallow” you specify per crawler which folders or pages are accessible. That way you keep internal search results or test pages out of the crawl process, so crawlers spend their time on important pages.

Robots.txt and indexing

Important: robots.txt blocks crawling, not necessarily indexing. A blocked page can sometimes still appear in the results if other pages link to it. If you really want to keep a page out of the index, use a noindex directive.

Common mistakes

A misplaced “Disallow” can accidentally block your entire site for search engines, with a big loss of traffic as a result. So always check your changes, and reference your sitemap in the file.

Does every site need a robots.txt?

Not strictly, but it gives you control over crawling. Without the file, crawlers may in principle visit everything.

Where is the file located?

In the root folder of your domain, at yourdomain.com/robots.txt.