UrlMapping to static files in Grails
Asked Answered
I

3

6

I want to map the static files sitemap.xml and robots.txt which a located in my web-app directory. The urls should be as follows:

http://www.mydomain.com/sitemap.xml 
http://www.mydomain.com/robots.txt

How do I have to set the url mapping to make these routes work?

Insufflate answered 31/7, 2013 at 14:37 Comment(0)
T
8

I use this mapping for robots.txt:

"/robots.txt" (view: "/robots")

And then have a grails-app/views/robots.gsp that contains the content for robots.txt. This way I can use <g:if env="..."> to easily have different content for different environments.

In order for this to work for a ".xml" extension, you need to change the Content Negotiation config.

grails.mime.file.extensions = false // disables the parsing of file extensions from URLs into the request format
Toolmaker answered 31/7, 2013 at 14:47 Comment(3)
Thanks. It works for robots, but not for sitemap.xml how would you do that?Insufflate
Does this solution has any any side effects?Insufflate
Automatic content negotiation based on file extensions won't happenToolmaker
U
12

The simplest way is to tell grails to ignore them in UrlMappings.groovy:

class UrlMappings {
    static excludes = ['/robots.txt', '/sitemap.xml']

    static mappings = {
        // normal mappings here ...
    }
}
Unlicensed answered 31/7, 2013 at 21:49 Comment(1)
This answer more accurately represents what the OP requested, although the selected answer provides a working alternate approach.Currajong
T
8

I use this mapping for robots.txt:

"/robots.txt" (view: "/robots")

And then have a grails-app/views/robots.gsp that contains the content for robots.txt. This way I can use <g:if env="..."> to easily have different content for different environments.

In order for this to work for a ".xml" extension, you need to change the Content Negotiation config.

grails.mime.file.extensions = false // disables the parsing of file extensions from URLs into the request format
Toolmaker answered 31/7, 2013 at 14:47 Comment(3)
Thanks. It works for robots, but not for sitemap.xml how would you do that?Insufflate
Does this solution has any any side effects?Insufflate
Automatic content negotiation based on file extensions won't happenToolmaker
C
0

It might also be helpful to to set up a nofollow to your Staging environment if you are using one. Not sure if there is a use case to having a staging site indexed.... so if you agree you may be able to use these steps to help block that.

If you are using Tomcat, set an environment variable such as NOFOLLOW=true --> see here for example: TOMCAT_OPTS, environment variable and System.getEnv()

Next as mentioned by @doelleri set the urlMappings

UrlMappings

//Robots.txt
"/robots.txt"(controller: 'robots', action:'robots')

Then use your robotsController to detect the environment variable you set on your staging tomcat.

RobotsController

def robots() {
    if (System.getenv('NOFOLLOW') == 'true') {
        def text = "User-agent: *\n" +
            "Disallow: /cgi-bin/ \n" +
            "Disallow: /tmp/ \n" +
            "Disallow: /junk/ \n" +
            "Disallow: /admin/ \n" +
            "Crawl-delay: 5 \n" +
            "Sitemap: https://www.example.com/sitemap.xml"

        render(text: text, contentType: "text/plain", encoding: "UTF-8")
    } else {
        render(status: 404, text: 'Failed to load robots.txt')
    }
}

robots.gsp

<%-- Content rendered from controller -> so leave blank :) --%> 
Clayborne answered 7/12, 2015 at 19:17 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.