Blocking Google from indexing your site

• December 14 2022
• 4 min. read

There are times when you don't want Google search to index specific pages on your site. It may be because these are private pages that, for some reason, are not password protected; or the site needs to be available to the client during development and you don't want to put it behind a password.

This post will present two ways of blocking Google crawlers from indexing your site: server response headers and client-side meta tags.

Using Server Response Headers #

The easiest way to add the headers is to do so in your server configuration. To disable search engine indexing globally you can add the following header to your server's configuration.

X-Robots-Tag: noindex

The specific way of doing this will depend on what server you're running.

For Apache servers, make sure that mod_headers is installed.

Headers can be added in the following places:

• The default server configuration
  • Adding it here makes the header global
• Virtual host configuration
• Directory
• Location

The header that will prevent Google from indexing is:

Header set X-Robots-Tag: "noindex"

Nginx setup is a little more complex.

add_header adds the specified field to a response header provided that the response code equals one of the following codes:

• 200
• 201
• 204
• 206
• 301
• 302
• 303
• 304
• 307
• 308

add_header directives are inherited from the previous configuration level if and only if there are no add_header directives defined on the current level.

If the optional always parameter is specified, the header field will be added regardless of the response code.

add_header can be added in the following contexts: http, server, location, if (when inside a location, but see If is Evil… when used in location context)

add_header X-Robots-Tag: "noindex";

Using meta tags #

If you have access to the server and are comfortable changing the configuration, then please use it. It's one less thing to remember and it'll work better in the long run.

But developers don't always have access to the server or are not comfortably changing the server's configuration.

We can still prevent indexing using meta tags in the head of the pages we want to crawlers to skip.

To prevent most search engines from indexing a page on your site, place the following meta tag into the head section of your page:

<meta name="robots" content="noindex">

To prevent only Google web crawlers from indexing a page:

<meta name="googlebot" content="noindex">

Note that googlebot is the name of the Google search engine crawler. It is not the only crawler that Google uses to crawl your content.

So what Google crawler do we want to stop indexing our content? #

Google uses multiple crawlers to navigate through your content. Most website developers and owners only deal with the Googlebot, the main Google crawler, and, usually, this is fine.

However, there may be times when there are specific kinds of items that you want to block without affecting the basic Googlebot settings.

In conclusion. Yes, it is possible to prevent crawlers in general and Google crawlers in particular from indexing your site and showing them in your results.

However, I'm not certain how long it'll take for the crawler to index your site after you remove the meta tag or the response header if it will do it at all.

Links and Resources #

• Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
• Block Search indexing with noindex
• Overview of Google crawlers (user agents)
• Googlebot. Google's main search crawler

Edit on Github