robots.txt content itself is indexed? [closed]

Question

The contents of my robots.txt file are actually itself indexed and show up in Google search results. It's only Google and not Yahoo for example.

I really think Google should understand not to index the contents of my robots file as it's only there to tell Google what not to index!

Am I missing something here?

Debbradebby

have you ever had a link, on your robot.txt file within the website? – Selig

No, I've had no link to the file – Debbradebby

Answer 1

That's normal. Mine was indexed a few months ago too. I think it's an issue with the crawler. edit: btw, you can delete the link on the webmaster tools website.

Outrigger

Answer 2

I wouldn't recommend disallowing the robots.txt file from itself. Instead you can use the X-Robots-Tag http header with a value of noindex which will prevent the search engines from Indexing the file.

This will allow them access to your robots.txt but prevent it from showing up in the search results.

Ass

Answer 3

I Think that disallowing your own robots.txt file (if possible) will result in all the other commands written there to be ignored by bots also.

Syrinx

If that was the case, putting 'Disallow: *' in the beginning could have the same problem. – Oquinn

Answer 4

You can add Disallow: /robots.txt to your robots.txt. I think it should avoid the robots.txt itself from being indexed.

Miscellanea

I will try this! Somewhat strange though to disallow something it is already reading. ;-) Thanks! – Debbradebby