Noindex, robots.txt and canonical: the differences

robots.txt, noindex and canonical are three instructions for search engines that are often confused. robots.txt is a file telling Google which parts of the site not to visit; noindex is an instruction on a page telling it not to show that page in results; canonical indicates which is the main version when several pages are near-identical. They all guide Google, but at different moments: before the visit, after the visit, when there are copies.

The comparison in one table

	robots.txt	noindex	canonical
What it is	A file at the site root	An instruction on the page	A link on the page
Tells Google	“Don’t visit here”	“Don’t show this page”	“The main page is this other one”
When it’s useful	Areas not worth crawling (endless filters, internal search)	Pages not needed in results (cart, account area, thank-you page)	Near-identical pages (filters, parameters, print versions)
Watch out	Doesn’t remove pages from results	Google must be able to visit the page	It’s a hint, not an order

What robots.txt is

It’s a text file that must sit at the site’s root, at /robots.txt (for example https://www.example.com/robots.txt). It tells search engine programs which paths not to visit. Example:

User-agent: *
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xml

The main rules:

Directive	What it does	Example
User-agent	Which program it addresses (* = all)	User-agent: Googlebot
Disallow	Path not to visit	Disallow: /cart/
Allow	Exception inside a blocked path	Allow: /cart/help/
Sitemap	Sitemap address	Sitemap: https://www.example.com/sitemap.xml
* and $	Any text / end of the address	Disallow: /*?sort=

Google reads at most the first 500 KiB of the file. An empty robots.txt, or Disallow: with no path, means “crawl everything”.

It’s used to avoid wasting Google’s visits on useless pages. It isn’t for hiding a page from results: if other sites link to it, it can still appear, without a description. And it isn’t protection: the file is public and anyone can read it, so don’t use it to hide private areas.

What noindex is

It’s an instruction in the page’s code, inside the meta robots tag:

<meta name="robots" content="noindex">

Google visits the page, reads the noindex and removes it from results. It’s the right choice for the cart, account area, post-order thank-you pages and internal search results.

For files that aren’t HTML pages, such as PDFs, noindex goes in the HTTP header sent by the server:

X-Robots-Tag: noindex

Other useful meta robots values: nofollow (don’t follow the page’s links), noimageindex (don’t index the images), max-snippet:0 (don’t show a text snippet).

A real example: the pages of Peacchy’s exercise app have a noindex, because they’re interactive exercises rather than pages to read; robots.txt, on the other hand, lets Google visit everything, so it can read that noindex.

The most dangerous mistake: a noindex added by mistake to important pages, for example after a theme change or when moving from a staging site to the live one. The pages disappear from Google within a few weeks.

What canonical is

When several addresses show almost the same content, for example /green-teas/ and /green-teas/?sort=price, you indicate which is the main version:

<link rel="canonical" href="https://www.example.com/green-teas/">

That way Google concentrates signals on a single page instead of splitting them across copies. Practical rules:

• use the full address (with https:// and domain), not a relative one;
• only one canonical per page;
• every “normal” page can have a canonical pointing to itself (self-referencing): it avoids copies created by parameters like UTMs;
• the canonical must point to a page that works (200 status), not to a redirect or a 404.

The canonical is a hint: Google almost always follows it, but may pick another page if signals conflict (for example if the sitemap and internal links point to a different address). In Search Console it shows as “Duplicate, Google chose different canonical than user”.

Wrong combinations

Combination	What happens	What to do
robots.txt blocks + noindex	Google can’t visit the page, doesn’t see the noindex and the page may stay in results	Remove the block from robots.txt until the page has dropped out
noindex + canonical to another page	Contradictory messages	Choose one of the two
canonical to a noindexed page	Google may index neither	Point the canonical to an indexable page
Staging site online without protection	Test copies end up on Google	Protect the staging environment with a password

How to check them

• Open /robots.txt in your browser to see the file.
• In Search Console, the robots.txt report shows the version Google read; URL Inspection shows whether the page has a noindex and which canonical Google chose.
• In the page source (right-click → View page source) search for noindex and canonical.

An example: Five Leaf

In Peacchy’s fictional shop two things happened after a theme change:

1. The herbal tea pages had a noindex by mistake: visits from Google dropped and the problem showed up in Search Console’s Pages report.
2. The filter pages (by price, by origin) had no canonical: Google found dozens almost identical to the category.

Fix: remove the noindex from the herbal teas and ask Google to recheck; add a canonical from the filters to the category.

How Google finds and stores pages is explained in how Google finds pages.

Sources

• Introduction to robots.txt (Google Search Central)
• Block search indexing with noindex (Google Search Central)
• How to specify a canonical URL (Google Search Central)