Does Google crawl AJAX content? [closed]

Question

On the home page of my site I use JQuery's ajax function to pull down a list of recent activity of users.

The recent activity is displayed on the page, and each line of the recent activity includes a link to the user profile of the user who did the activity.

Will Google actually make the ajax call to pull down this info and use it in calculating page relevancy / link juice flow?

I'm hoping that it does not because the user profile pages are not very Google index worthy, and I don't want all those links to the User profile pages diluting my home page's link juice flow away from other more important links.

Kamikamikaze

Answer 1

No, it will not crawl AJAX content by default.

https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics has instructions on how to make AJAX content crawlable, but those are explicit steps you need to take, it isn't automatic

Ansilme

That link says: "This recommendation is officially deprecated as of October 2015". – Federalese

I've edited the answer to use an up-to-date link: developers.google.com/search/docs/crawling-indexing/javascript/… – Horde

Answer 2

Maybe. If you want to guarantee Google won't spider your JSON responses, put them in robots.txt. That isn't security, though; indeed, it's the first place a cracker will look for "interesting" pages. And other crawlers may ignore it.

Mcmullen

hmm, maybe i could do that with the robots.txt. That wouldn't be considered blackhat SEO would it? Since the only reason I'm restricting it is to sculpt my pagerank flow? – Kamikamikaze

It's not "blackhat SEO" to put a page in robots.txt. But this won't guarantee that the googlebot won't see that data if you include it in a non-excluded page. Generally, any attempt to make the page substantively different to the googlebot from what a real user sees is not allowed. But asynchronous content is a special case, since 1) it's common, and 2) Google is still figuring out how to deal with it. I suspect that sooner or later it will be indexed. – Mcmullen

Answer 3

Yes, Google crawls dynamic content created using javascript. It can recognize the DOM after loading, including modifications to the title tag. It can also follow links created with the onclick event handler.

Read about it here: http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157

It will respect nofollow however, for your profile links.

Postage

Answer 4

Google now has a way for you to enable crawling on AJAX pages. If your links contain "#!", then google will change that to "?_escaped_fragment_=" and request that document from your server. However, when it shows that page in the search results, it will show the original URL with the "#!".

http://code.google.com/web/ajaxcrawling/docs/learn-more.html

Johm

That link leads to a page that says: "This recommendation is officially deprecated as of October 2015". – Federalese

Answer 5

Google is definitely crawling content in our page that is only referenced within an AJAX request.

I'm wondering if Google is going through the page source, and looking for potentially valid URLs, and testing to see if they've got content.

Here is what our request looks like... which might offer some insight into what's going on.

'$(document).ready(function() {
    $("#theDiv").block({ message: 'Getting latest content...' });
    $.ajax({
        url: '/content/pages/articles?count=4&part=true',
        success: function(data) {
            $('#theDiv').html(data);
            $("#theDiv").unblock();
        }
    });
});`

Meingolda

This answer is pretty old. But, it should be obvious to everyone now that Google is basically running and rendering your page in an actual web browser. They are definitely executing AJAX requests. – Meingolda

Answer 6

Google itself doesn't crawl AJAX content, but it proposed a scheme to make AJAX content crawl. I wrote a an article to explain that how it works. It also includes very simple code to demonstrate Google Crawling Scheme for AJAX content. Find it here: http://www.tayyabali.com/seo-ajax-content-asp-net-code-to-crawl-ajax

Iorgo

Answer 7

Here is some info on AJAX crawling from the makers of JQuery Address plugin.

Yount

Answer 8

For Ruby users there is a gem which handles this relatively painlessly.... google_ajax_crawler Github repo here, writeup about using it for backbonejs at thecodeabode

Basile

Answer 9

in the last month google has stopped to support ajax! https://googlewebmastercentral.blogspot.fr/2015/10/deprecating-our-ajax-crawling-scheme.html

Hippy

It did not stop supporting AJAX. Instead, it now does it automatically by rendering the page the way a browser would. Google is saying that you no longer have to jump through the extra hoops it asked for back in 2009. – Malmsey