Where to put robots.txt in tomcat 7?

Question

I'm using Tomcat 7 to host my application. I've used a ROOT.xml file under tomcat-home\conf\Catalina\localhost

<Context 
  docBase="C:\Program Files\Apache Software Foundation\Tomcat 7.0\mywebapp\MyApplication" 
  path="" 
  reloadable="true" 
/>

This is to load my webapp in the root context.

But now I'm confused as to where to put the robots.txt and sitemap.xml files. When I put in under C:\Program Files\Apache Software Foundation\Tomcat 7.0\mywebapp\MyApplication, it doesn't show up.

I've also tried placing it inside C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ROOT , just in case. But nothing works. I'm still getting 404 not found. Can anyone please guide.

p.s. My web application runs fine. It's just that my robots.txt file is unreachable.

EDIT

My web.xml file:

    <filter>
        <filter-name>springSecurityFilterChain</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy
        </filter-class>
    </filter>

    <filter-mapping>
        <filter-name>springSecurityFilterChain</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

Erotica

Answer 1

It should be in your context directory

mywebapp\MyApplication

just where you will have your web-inf directory,index.html etc.

UPDATE:

As per our discussion all urls are handled by Spring MVC DelegatingFilterProxy , so you need to exclude *.txt from this filter somehow, may be by extending DelegatingFilterProxy and configuring that filter in web.xml

SOLUTION: I can server static content through this hack in Spring MVC:

<mvc:resources mapping="/robots.txt" location="/robots.txt" order="0"/>

Saved me a ton of headache. :) Writing it for anyone else to benefit from.

Grandchild

I've put it there, but it still gives me 404. – Erotica

what is the link where you get 404? is it http://serverip:port/robots.txt ? if so http://serverip:port/index.html is it loading – Grandchild

Yes, serverip:port loads. I've my home page file under WEB-INF/jsp/home.jsp and when I goto serverip:port, it loads. – Erotica

can you post your web.xml to see if you have any special mapping – Grandchild

Let me update the question, so that the web.xml is more viewable. – Erotica

ok problem is <filter-mapping> <filter-name>springSecurityFilterChain</filter-name> <url-pattern>/*</url-pattern> </filter-mapping> all requests are routed through DelegatingFilterProxy – Grandchild

instead of /* can you add specific types like .html/.htm/*.do etc, so that *.txt is not filtered – Grandchild

let us continue this discussion in chat – Erotica

its not a hack, its the normal method for serving static content – Taimi

I followed your answer but I cannot access it yet. My question is here #32302965 – Marvismarwin

Answer 2

Just put the robots.txt in Tomcat_DIR/webapps/ROOT/

Nannienanning

Answer 3

I've got the same issue: I doesn't meter where I put my robots.txt or configure <mvc:resources mapping="/robots.txt" location="/robots.txt" order="0"/> each request to robots.txt returned 404 status.

Problem here is that I use both Spring MVC and Backbone.js MVC on client side, which has its routing. So I couldn't adjust <url-pattern></url-pattern> to serve both MVCs.

The only quick and ugly solution I found here is to process requests to /robots.txt as a request in Spring MVC

@RequestMapping(value = "/robots.txt", produces = {"text/plain"}, method = RequestMethod.GET)
@ResponseBody
public String getRobotsTxt() {
    return "User-agent: *" + 
   "\n" + "Allow: /js/views/*.js" + 
   "\n" + "Allow: /js/*.js" + 
   "\n" + "Allow: /css/*.css" + 
   "\n" + "Allow: /css/*.png" + 
   "\n" + "Disallow: /cgi-bin" + 
   "\n" + "Sitemap: http://blabla/sitemap.xml";
}

Caras