https://developers.google.com/+/web/snippet/ Is this a crawler bot [closed]

Question

I see significant traffic coming from this Bot when we send SMS to users. I am trying to get more details about this bot. Any pointers describing what it does, how important it is, and can we block it would be appreciated. The complete user agent is

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/)

Vaules

I voted to close this question because it is not a programming question and it is off-topic on Stack Overflow. Non-programming questions about your website should be asked on Webmasters. In the future, please ask similar question there. – Leyte

Answer 1

Google's SMS app on Android uses this user agent when it crawls the shared URL to generate a preview inside the chat. This is usually a good thing because your customers will see the preview image and page title, rather than just a link. So I would not recommend blocking it.

Ariadne

When we run an SMS campaign, for a moment this causes a significant increase in traffic to servers. For some reason, this also bypasses the CDN cache, maybe because of user-agent. Any idea, how to handle this? – Vaules

Maybe a bit late, but possibly getting lots of traffc from differnt Googel servers. Such that many of the servers are hitting different CDN edge nodes, so the hit rate is bad. As for dealing with it maybe your CDN has a way to share the cache between nodes – Disquiet

Answer 2

You can block google bots on your nginx configuration. Inside of the server{} section, add this:

if ($http_user_agent ~* "developers\.google\.com/\+/web/snippet") {
    return 403;
}

Container

Any idea if this bot is of any importance? – Vaules

I think they are used to index searches on google, but I'm not sure. I had to block it because it was overloading my server. – Container

@Container - did you find a good way to achieve this? I am facing a similar problem – Mancuso

Answer 3

This bot is indeed used by some SMS Android applications to fetch a preview of the we page.

Phones using it include (but it has changed over time) Xiaomi, Huawei, Oppo.. For some websites, it is important to block it. For instance, my app is sending a confirmation link, having it clicked by the preview is messing up our stats and follow-up of who wants that.

To block it, you can use the folowwing .htaccess rule (if you use Apache) :

RewriteEngine on

RewriteCond %{HTTP_USER_AGENT} ^.developers.google.com/+/web/snippet.$ [NC]

RewriteRule "^.*$" - [F,L]

Herren

Answer 4

Anyone looking for ASP.NET-MVC solution:

Create a helper class to gather a list of all the known bots. For the above case, the bot is Google-PageRenderer. The other list is of the well-known bots:

using System;
using System.Collections.Generic;
using System.Linq;

namespace MyProject.Helpers
{
    public static class BotDetectionHelper
    {
        private static readonly HashSet<string> Bots = new HashSet<string>(StringComparer.Ordinal)
        {
            "Google-PageRenderer",
            "Googlebot",
            "Bingbot",
            "Slurp",
            "DuckDuckBot",
            "Baiduspider",
            "YandexBot",
            "Sogou",
            "Exabot",
            "facebookexternalhit",
            "LinkedInBot",
            "Twitterbot",
            "Pinterestbot",
            "WhatsApp",
            "DotBot",
            "spbot/",
            "James BOT",
            "baidu",
            "Baidu",
            "/bot",
            "semantic-visions.com",
            "spider",
            "lipperhey",
            "linkdexbot/",
            "MJ12bot/",
            "Lipperhey-Kaus-Australis/",
            "BDCbot",
            "AhrefsBot",
            "SemrushBot",
            "Alexa",
            "Uptimebot",
            "Crawl",
            "Spider",
            "PageSpeed",
            "ZoominfoBot",
            "Adidxbot",
            "BLEXBot",
            "SEOkicks",
            "BlackWidow",
            "BotALot",
            "Buddy",
            "BuiltWith",
            "Curl",
            "DISCo",
            "Dotbot",
            "Feedfetcher-Google",
            "Geekbot",
            "GrapeshotCrawler",
            "GT::WWW",
            "HTTP::Lite",
            "HubSpot",
            "ia_archiver",
            "Jetbot",
            "JetBrains Omea Reader",
            "Mechanize",
            "NetcraftSurveyAgent",
            "Nutch",
            "Outbrain",
            "Python-urllib",
            "rogerbot",
            "ShowyouBot",
            "SiteExplorer",
            "Slackbot",
            "Teoma",
            "Twingly Recon",
            "Via",
            "Wget",
            "Xenu Link Sleuth",
            "ZmEu"
        };

        public static bool IsBotAgent(string cBot) =>
            Bots.Any(bot => cBot.IndexOf(bot, StringComparison.Ordinal) != -1);
    }
}

Then create a method in your Controller class:

private bool BotDetectionResult()
{
    bool isBot = false;
    string userAgent = Request.UserAgent;

    try
    {
        if (userAgent != null)
        {
            isBot = BotDetectionHelper.IsBotAgent(userAgent);
        }
    }
    catch (Exception ex)
    {
      //Log exception
    }

    return isBot;
}

Usage:

bool detectBot = BotDetectionResult();
if (!detectBot)
{
  //Send to database etc.
}

If the bots list can be updated to include other well known bots, then please feel free to edit the answer and add your entries.

Yam

The question was about a specific bot, not about identifying a wide range of bots. – Leyte

@StephenOstermiller Yes that is what I have answered. I have added the specific bot name only. Please review carefully before jumping to a conclusion. It is not a good thing (especially when you are a moderator) to just scrape through an answer and make judgement on it. With great power comes great responsibility. – Yam

Really? The quetion is about what Google (+https://developers.google.com/+/web/snippet/) is. This answer doesn't address that. Rather it has a method to identify bots by user agent. The list of bots identified includes Googlebot and DuckDuckBot but notably DOESN'T actually include a user agent of just Google. – Leyte

@StephenOstermiller O.P question: and can we block it would be appreciated ? My answer: For the above case, the bot is Google-PageRenderer. And after that I have given steps on how to block it and not only identify it – Yam