How Google finds pages and how to get your site indexed

Google finds pages in three steps: crawling (a program, Googlebot, visits pages by following links), indexing (Google reads the page and decides whether to store it in its archive, the index) and ranking (when someone searches, it picks the most useful pages and puts them in order). To get a site indexed on Google you need to make pages easy to find, submit a sitemap in Search Console and avoid instructions that exclude them.

Every time you search for something, Google answers in under a second. It doesn’t search the whole web at that moment: it searches its archive, built beforehand.

1. Crawling: finding pages

A Google program (called Googlebot) visits pages and follows links, from one page to the next. That’s why internal links matter: a page nobody links to is hard to find.

A sitemap helps too: a file listing the site’s pages, submitted to Google in Search Console. A sitemap can contain at most 50,000 URLs or be 50 MB (uncompressed): bigger sites use several, grouped in a sitemap index.

The robots.txt file tells Googlebot which parts of the site not to visit. Careful: blocking a page in robots.txt does not remove it from Google, it only stops Google reading it.

For the vast majority of sites Google visits every page without trouble. The so-called crawl budget (how many pages Google crawls in a given time) only matters, according to Google, for very large sites: over one million pages that change often, or over 10,000 pages that change daily.

2. Indexing: storing pages

Google reads the page content and decides whether to store it in its index, the archive results come from. It may decide not to, for example if:

• the page asks not to be indexed (a noindex);
• it’s almost identical to another page (a duplicate);
• it has little useful content;
• it’s behind a password or returns an error (404, 500);
• Google can’t read it properly, for example because the text only appears after lots of scripts.

How these instructions are controlled is explained in noindex, robots.txt and canonical.

3. Ranking: putting pages in order

When someone searches, Google picks the pages in its index that answer best and puts them in order. It looks at many signals:

• whether the page matches the searcher’s intent (see search intent);
• whether the content is helpful and trustworthy (the E-E-A-T criteria);
• whether other sites cite it with a link;
• the searcher’s language and location;
• whether it opens well on a phone and without waiting (see Core Web Vitals).

How to get your site indexed on Google in 7 steps

1. Check the site is public: no password, no “under construction” mode blocking search engines.
2. Add the site to Google Search Console and verify the property.
3. Submit the sitemap from the Sitemaps section.
4. Link every important page from other pages on the site (menus, articles, categories).
5. Remove accidental noindex tags and check robots.txt.
6. Use URL Inspection for new pages and click “Request indexing”.
7. Wait and check the Pages report: a new page can be indexed in a few days or a few weeks. Google doesn’t guarantee that every page will be indexed.

Why isn’t my site showing up on Google?

Symptom	Likely cause
No pages on Google	New site not yet crawled, or noindex across the whole site
Some pages missing	No internal links, near-identical content, noindex on a page template
Pages are there but low down	Strong competition, less useful content than others, wrong keyword
They suddenly disappear	Server errors, wrong redirects, a theme or platform change

To check whether a page is indexed you can also search Google for site:example.com/page, but the precise answer is in Search Console’s URL Inspection.

An example

At the fictional shop Five Leaf, after a site theme change, the herbal tea pages were crawled but not indexed: the new theme had added a noindex by mistake. The pages existed, Google visited them, but didn’t store them. The problem wasn’t ranking: it was step two.

To see where a real site’s pages stand, use Google Search Console.

Sources

• How Google Search works (Google Search Central)
• Build and submit a sitemap (Google Search Central)
• Large site owner’s guide to managing your crawl budget (Google Search Central)