Precedence of X-Robots-Tag header vs robots meta tag

Question

I've placed the following Header in my vhost config:

Header set X-Robots-Tag "noindex, nofollow"

The goal here is to just disable search engines from indexing my testing environment. The site is Wordpress and there is a plugin installed to manage per-page the meta robots settings. For example:

<meta name="robots" content="index, follow" />

So my question is, which directive will take precedence over the other since both are being set on every page?

Carnify

Great question. My suspicion is that, since index,follow are the default values, any noindex,nofollow directives that show up, whether in header or meta, will trump it. But I'd like to hear a definitive answer. Adding bounty... – Callahan

Answer 1

I am not sure if a definitive answer can be given to the question, as the behavior may be implementation-dependent (on the robot side).

However, I think there is reasonable evidence that X-Robots-Tag will take precedence over <meta name="robots" .... See :

One significant difference between the X-Robots-Tag and the robots meta directive is:

• X-Robots-Tag is part of the HTTP protocol header.
• <meta name="robots" ... is part of the HTML document header.

Therefore the the X-Robots-Tag belongs to HTTP protocol layer, while <meta name="robots" ... belongs to the HTML protocol layer.

As they belong to a different protocol layer, they will not be parsed simultaneously by the (robot) client getting the page: The HTTP layer will be parsed first, and the HTML in a later step.

(Also, it should be noted that X-Robots-Tag and <meta name="robots" ... are not suppported by all robots. Google and Yahoo/Bing suppport both, but according to this some support only <meta name="robots" ..., others support neither.)

Summary :

• if supported by the robot, X-Robots-Tag will be processed first ; restrictions (noindex, nofollow) apply (and <meta name="robots" ... is ignored).
• else, <meta name="robots" ... directive applies.

Wheelwork

The way this is explained makes perfect sense to me and I would have to agree with the facts. Can anyone else confirm this? – Carnify

In my case (http header contains X-Robots-Tag as "noindex, nofollow" AND html document head contains robots meta tag as "index,follow"), I was expecting that page would be removed from search results. BUT after waiting 3 days it seems doesn't work. Below answer from @Tremain may be correct. So I will modify the html head to make it compatible with http header X-Robots-Tag and will observe the results. – Demure

Answer 2

Just an update to Dan's experience, I also have both the

Header set X-Robots-Tag "noindex, nofollow"

and

<meta name="robots" content="index, follow" />

on my one of my Wordpress sites, and a check in Google Search Console confirmed that the noindex in X-Robots-Tag is taking precedence as the pages have been crawled and but aren't indexed. So the logic in the correct answer is indeed, correct.

Mo

Answer 3

In my recent experience, when Google sees mixed-messages it prefers positive action by default - ie - it favours indexing - meanwhile will flag the issue as a critical error/warning in your webmaster tools console if you have one.

see your site's status in google here: https://www.google.com/webmasters/

see you site's status in bing here: http://www.bing.com/toolbox/webmaster (note that yahoo search is now powered by bing)

Google takes this positive-by-default action because lots of site owners unwittingly have a dodgy cms semi-blocking robots and we know how google loves to accumulate as much data as it can - any excuse!

if the technical settings are erroneous they're liable to be totally disregarded, and we know how search engines index and follow by default when no settings are specified.

Tremain