Where to put robots.txt file? [closed]

Question

Where should put robots.txt?

domainname.com/robots.txt

or

domainname/public_html/robots.txt

I placed the file in domainname.com/robots.txt, but it's not opening when I type this in browser.

alt text http://shup.com/Shup/358900/11056202047-My-Desktop.png

Mendel

Answer 1

Where the file goes in your filesystem depends on what host you're using, so it's hard for us to give a specific answer about that.

The best description is: put it wherever the index.html (or index.php or whatever) file is that represents your homepage. If that's domainname/public_html/index.html, for example, put it in domainname/public_html/robots.txt.

Hyehyena

thanks it's working now i placed at domainname/public_html/index.html – Mendel

That's not correct. It has to be in the root directory of the web server, which is different if your home page is in a subdirectory. – Agog

By "homepage" here I mean the proper homepage for the domain. Some robots will also read robots.txt files from subdirectories, but that's not as reliable as "the" robots.txt at a website's root. – Hyehyena

I followed your answer but I cannot access it yet. My question is here #32302965 – Realm

Answer 2

i think the better way to describe it is to have it in the root web folder of your domain... so http://example.com/robots.txt you can also put your sitemap.xml in the root or refer to it with a Sitemap: http://example.com/fldr/smap.xml line in your robots.txt.

dont forget: you can use Google Webmaster Tools to check to make sure you haven't restricted anything you didnt mean to(you also get to see queries and links woohoo!).

suggestion: id consider using the <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> if possible because you will still earn linkjuice for links on the page but it wont show up in googles index while a robots.txt directive can leave a plain url with do description in SERPs but will loose all value of links pointed to it because its robots.txted out (its ranking b/c of anchor text so get credit for it)

Empyreal

Answer 3

In the root of your web directory (where you put the files that show up on your website)

Handbarrow

Answer 4

In this case you should put it in domainname/public_html/robots.txt, as the public.html folder is where your index file will be.

Skivvy