Why does Googlebot crawl for /mobile/* and /m/* pages that are not referenced anywhere?

Question

Since the end of may, I have a lot of new 404 errors in the Smartphone Crawl Errors page in Webmaster Tools / Google search console. All of them starts with /m/ or /mobile/, none of which are existing nor linked to anywhere on the site.

For example, I have a 404 error for http://www.example.com/mobile/foo-bar/ and http://www.example.com/m/foo-bar pages. According to the Search Console, those page are linked in the existing page http://www.example.com/foo-bar/, but they are not.

Is Googlebot deciding on its own to look for a mobile version of every page ? Can I disable this behavior ? Is this because my site is not mobile-friendly yet (a problem for which I received another warning message from Google).

Intercut

Same exact issue for me today. Suddenly lots of 404s for pixelscrapper.com/m/... and pixelscrapper.com/mobile/... even though I do not have any pages on those paths, and no links to such paths. Only on the "Smartphone" tab of Webmaster Tools. I think Google is indeed "guessing" at these paths when crawling as a mobile device. – Cloaca

Same issue here, starting 3 weeks ago and increasing. /m/... /mobile/... 404 errors. Seems Google is trying to detect mobile version on non-mobile sites. Well, does not really make sense to me :) Think, I'll block those with robots.txt – Fontes

Answer 1

As @Jonny 5 mentioned in a comment, this seems to be happening as a result of Google guessing that you may have a mobile version of your site in the /m and/or /mobile directories. From what I have read, they will only try those directories if they decided that the pages they initially indexed were not mobile-friendly/responsive. More info on this behavior can be found in these Google Product Forum threads:

• https://productforums.google.com/forum/#!topic/webmasters/k3TFeCkFE0Q
• https://productforums.google.com/forum/#!topic/webmasters/56CNFxZBFwE

Another helpful comment came from @user29671, who pointed out that your website does in fact have some URLs with /m and /mobile indexed. I found that the same was true for my website, so this behavior may also be limited to sites that Google has (for whatever reason) indexed a /m and/or /mobile URL for. To test if this is true for your site, go to the following URLs and replace example.com with your website's domain:

• https://www.google.com/search?q=site:example.com/m/&filter=0
• https://www.google.com/search?q=site:example.com/mobile/&filter=0

As far as preventing this goes, your best bet is either creating a mobile-friendly version of your site or redirecting /m and /mobile pages back to the originals.

Unclasp

It seems to me that Google is requesting the /m and /mobile urls by default. It has nothing to do whether your site is mobile friendly or not. I get that /m and /mobile crawl errors for each new site i add to WMT, while google.com/webmasters/tools/mobile-friendly is telling me the site is mobile friendly indeed. – Phira

Interesting. Does it say the exact pages which it is appending /m and /mobile to are already mobile friendly? – Unclasp

Not sure if I understood your question correctly, though It is just one responsive site for every device, the /m and /mobile returns a 404 since they don't excist. Neither does WMT shows any referrals pointing towards those /m and /mobile. For reference, this is one of those sites psycholoog.site – Phira

Yes, that question was a bit unclear. Do you have an example of a /m or /mobile URL that Google tried to check? – Unclasp

Interesting. Yes, if those two pages were checked, then I think you are correct - Google may sometimes check for /m and /mobile even if the page in question is already responsive/mobile-friendly. – Unclasp

Answer 2

You could block those directories in your robots.txt, but that's a bit of a workaround. The better option would be to figure out where exactly Googlebot is picking up those URLs from.

If you shared an example page URL where Google says you have links to the /mobile pages, I could look at it and figure out where that's being picked up.

And no, Google doesn't just invent directories to crawl on the off-chance that you might have snuck in a mobile page randomly :)

Larson

An example: the page belial.fr/leigh-brackett is supposed to link to belial.fr/m/leigh-brackett and belial.fr/mobile/leigh-brackett according to Search Console – Intercut

Interesting... I don't see any direct links on the page itself, that's true. Have you currently or in the past had any server-side redirects for mobile users? – Larson

I never did anything related to /m/ or /mobile/ urls, nor link nor redirection, that's why I'm puzzled by them appearing in Google search console and thought it's some kind of standard. – Intercut

Actually, google has got a history of guessing urls in certain circumstances and returning 404 for non-existent urls is the proper way. It will be noted in webmaster tools, no big deal. – Vallievalliere

and you have got urls with /mobile/ and /m/ on your sub-domains : google.co.uk/… – Vallievalliere

Answer 3

I am experiencing the same issue since December 2016. Googlebot is constantly trying to crawl my website pages with the /m/ and /mobile/ prefixes.

All those urls cause the 404 errors and get listed in Google Webmaster Tools as errors.

The automatic email was received from GWT on January 2nd, 2017 stating

Googlebot for smartphones identified a significant increase in the number of URLs on http://example.com that return a 404 (not found) error. If these pages exist on your desktop site, showing an error for mobile users can be a bad user experience. This misconfiguration can also prevent Google from showing the correct page in mobile search results. If these URLs don't exist, no action is necessary.

This is done by a mobile crawler:

*Ip: 66.249.65.124

Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1)

Browser: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1)*

You are not alone, therefore. Take it easy. It's a Google bug :)

Mean

Answer 4

As for redirecting /m and /mobile pages back to the originals, here's a snippet for nginx:

location /m/ {
     rewrite ^/[^/]+(/.*)$ $1 permanent;
}

location /mobile/ {
    rewrite ^/[^/]+(/.*)$ $1 permanent;
}

One can also redirect everything to the root:

location /m/ {
    return 301 $scheme://$host/;
}

location /mobile/ {
    return 301 $scheme://$host/;
}

Boxing