What is robots.txt?

Robots.txtis a text file at the root of your website that tells search engines which parts they can and cannot crawl.It controls the behavior of crawlers and helps to save crawl budget on unimportant pages.It is a basic tool within the technical SEO of any website.What does

robots.txt do?With

rules such as “Allow” and “Disallow” you indicate per crawler which folders or pages are accessible.For example, you can keep internal search results or test pages outcrawlingcrawl process, so that crawlers spend their time on important pages.Robots.txt and indexingImportant:

robots.txt blocks

crawling, not indexing per se.A blocked page canindexingstill appear in the results when linked to it.If you really want to keep a page out of the index, use a noindex instruction.Common mistakesA misplaced “Disallow” can

accidentally block

your entire site from search engines, causing major traffic loss.Therefore, always check changes and refer to your sitemap in the file.Does every site need a robots.txt?Not strictly, but itsitemapyou

control over crawling.Without a

file, crawlers are essentially allowed to visit everything.Where is the file located?In the root folder

of your domain, accessible

via uwdomein.nl/robots.txt.