What is crawling?

Crawling is the process in which search engines use automated bots (crawlers) to scour the web and discover and read pages. The crawler follows links from page to page and collects the content that can then be indexed. Without crawling, Google does not know your pages exist.

How does crawling work?

A crawler, such as Googlebot, starts at known pages and follows the links it comes across. That is how new and changed pages are discovered. A good internal linking structure and an up-to-date sitemap help crawlers move through your site efficiently.

Steering crawling

With the robots.txt file you specify which parts of your site may or may not be crawled. That stops unimportant pages from eating up crawl budget unnecessarily. This is part of technical SEO.

Crawling versus indexing

Crawling is discovering and reading a page; indexing is storing it in the search index. A page has to be crawled before it can be indexed and shown.

What is crawl budget?

It is the number of pages a search engine crawls on your site within a certain period. Efficient crawling matters most for large sites.

How often does Google crawl my site?

That depends on the size, authority and update frequency of your website.