Need to block subdomain using robots.txt which is on same directory level

Question

I have one problem

I have domain name for example www.testing.com and new.testing.com so i do not want to new.testing.com display in any search engine. I have added one robots.txt to the new.testing.com. And both site has same parent directory

--httpdoc
----testing.com
----new.testing.com

So i want to know that can i handle both site using one robots.txt of testing.com is it possible???

Please suggest me solution if possible.

Amontillado

Answer 1

The best thing you could do is to add separate robots.txt files. Put one in each directory. You should have

• testing.com/robots.txt and
• new.testing.com/robots.txt

After adding robots.txt file to new.testing.com, you should add following code to keep search engines away.

User-agent: *
Disallow: /

Bacciferous

if domain and subdomain same directory how can disallow only subdomain? disallow: /new.domain.com/ that true? or have any another idea? please help... – Ahimsa

Answer 2

RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robots\.txt$ robots-subdomain.txt

Then add the following to /robots-subdomain.txt:

User-agent: *
Disallow: / 

Uncut

Answer 3

Following rules inside works for me,

RewriteEngine On
RewriteCond %{HTTP_HOST} ^subdomain\.maindomain\.com$ [NC]
RewriteRule ^/robots.txt$ /nobots.txt [L]

And add 'nobots.txt' to root directory as follows,

User-agent: *
Disallow: / 

Okay