<noindex> tag for Google

Question

I would like to tell Google not to index certain parts of the page. In Yandex (russian SE) there's a very useful tag called <noindex>. How can it be done with Google?

Puncture

Answer 1

You can prevent Google from seeing portions of the page by putting those portions in iframes that are blocked by robots.txt.

robots.txt

Disallow: /iframes/

index.html

This text is crawlable, but now you'll see 
text that search engines can't see:
<iframe src="/iframes/hidden.html" width="100%" height=300 scrolling=no>

/iframes/hidden.html

Search engines cannot see this text.

Instead of using using iframes, you could load the contents of the hidden file using AJAX. Here is an example that uses jquery ajax to do so:

his text is crawlable, but now you'll see 
text that search engines can't see:
<div id="hidden"></div>
<script>
    $.get(
        "/iframes/hidden.html",
        function(data){$('#hidden').html(data)},
    );
</script>

Samualsamuel

Note that the AJAX part is no longer correct. Most search engines evaluate JavaScript and execute XHR calls. – Erenow

If you are all loaded by Ajax is disallowed by JavaScript, search engines still won't be able to see it even if they execute JavaScript in general. – Samualsamuel

Answer 2

According to Wikipedia¹, there are some rules some spiders follow:

<!--googleoff: all-->
This should not be indexed by Google. Though its main spider, Googlebot,
might ignore that hint.
<!--googleon: all-->

<div class="robots-nocontent">Yahoo bots won't index this.</div>

<noindex>Yandex bots ignore this text.</noindex>
<!--noindex-->They will ignore this, too.<!--/noindex-->

Unfortunately, they could not agree on a single standard it seems – and to my knowledge, there's nothing to keep all spiders off...

The googleoff: comment seems to support different options, though I'm not sure where there's a complete list. There's at least:

• all: completely ignore the block
• index: content doesn't go into Google's index
• anchor: anchor text for links will not be associated with the target page
• snippet: text will not be used to create snippets for search results

Note as well that (at least for Google) this will only affect the search index, not the page ranking etc. Furthermore, as Stephen Ostermiller correctly pointed out in his comment below, googleon and googleoff only work with the Google search appliance and have no effect on normal Googlebot, unfortunately.

There's also an article on the Yahoo part² (and an article describing that Yandex also honors <noindex>⁶). On the googleoff: part, also see this answer, and the article I took most of the related information from.³

---

Additionally, Google Webmaster Tools recommend using the rel=nofollow attribute⁴ for specific links (e.g. ads or links to pages not accessible/useful to the bots, such as login/signup). That means, the HTML a rel Attribute should be honored by the Google bots – though that's mainly related to page rank, not to the search index itself. Unfortunately, it seems there's no rel=noindex^5,7. I'm also not sure if this attribute could be used for other elements as well (e.g. <DIV REL="noindex">); but unless crawlers honor "noindex", that wouldn't make sense either.

---

Further references:

• How to Noindex parts of a web page?
• Excluding crawler from sections of pages (Spiderline crawler; you see, other crawlers might use other proprietary markers (see also AddSearch crawler). I wish they simply made REL="noindex" a standard instead to be used with any HTML tag such as DIV/SPAN/P/A!)
• Preventing Google from indexing the contents of a div by reversing the string
• Methods for preventing search engines from indexing irrelevant content on a page

---

¹ Wikipedia: Noindex
² Which Sections of Your Web Pages Might Search Engines Ignore?
³ Tell Google to Not Index Certain Parts of Your Page
⁴ Use rel="nofollow" for specific links
⁵ Is it a good idea to use <a href=“http://name.com” rel=“noindex, nofollow”>name</a>?
⁶ Using HTML tags — Yandex.Help. Webmaster
⁷ existing REL values

Intercollegiate

googleoff and googleon only work with the Google search appliance and have no effect on normal Googlebot – Samualsamuel

@StephenOstermiller true, I figured that meanwhile as well. Thanks for pointing out, I completely forgot to update that here! – Intercollegiate

Answer 3

No, Google does not support the <noindex> tag. Virtually no one does.

Premium

Not in any way that Google approves of: webmasters.stackexchange.com/questions/16390/… – Premium

"Virtually noone" includes at least Yandex, see my answer. But who really cares about that one, is another question. – Intercollegiate

Answer 4

I had the same issue, the solution is to use data-nosnippet.

<p><span data-nosnippet>This text won't show in google results</span></p>

Doubleripper

data-nosnippet doesn't stop Google from indexing the content, it only means that they won't use that content for the snippet in their search results. Google will still index that content and use it when evaluating the page ranking, and they will still display it in their cached version of the page for everyone to see. – Bonnee

Answer 5

Create a robots.txt file at your root level and insert something like the following:

Block Google:

User-agent: Googlebot
Disallow: /myDisallowedDir1/
Disallow: /myDisallowedPage.html
Disallow: /myDisallowedDir2/

Block all bots:

User-agent: *
Disallow: /myDisallowedDir1/
Disallow: /myDisallowedPage.html
Disallow: /myDisallowedDir2/

A handy robots.txt generator:

http://www.mcanerin.com/EN/search-engine/robots-txt.asp

Corrianne

teslasimus doesn't want to block the whole page, only "certain parts". – Kingwood

good point, my answer can be used along with the iframe solution proposed above – Corrianne