Angular2 App: Fetch as Google doesn't load page content

Question

I am working on Angular2 based web app. I used Angular CLI to generate app and then to build it for prod. I have hosted website on AWS S3 & Cloudfront. When I use 'Fetch as Google' tool from the webmaster, it shows only Loading....

Isn't Googlebot able to crawl my website?

Sangria

Is your website frontend-only? If you don't have a backend to serve the file you could encounter this issue. – Fireside

I have used Angular2 for front-end and is served via AWS cloudfront. Back-end is nodejs and hosted on different sub-domain. Angular2 app makes API requests to nodejs app to render content. – Sangria

Is your frontend being served by any backend? or is it a bare index.html sitting on your domain? – Fireside

No, the frontend is not served by any server side language. It is hosted on AWS S3 and served by Cloudfront. – Sangria

Cross posted: webmasters.stackexchange.com/questions/105098/… – Carley

Answer 1

had a similar issue. I believe Google-Bot do not support modern JS. I simply activated all shims recommended by angular.io see https://angular.io/docs/ts/latest/guide/browser-support.html and added in the script header:

<script src="https://cdnjs.cloudflare.com/ajax/libs/core-js/2.4.1/shim.min.js">
</script>

If you really need all of them I'm not sure, but I need support for older IE anyway.

If this works for all Searchbots is very unclear, if you have to support them you can try https://prerender.io/. However they are also just able to render it with the shims

Hope this helps.

Amaranthine

This ended up being the resolution to my problem. I can't thank you enough for sharing this solution. I used the cdn link you shared, but I'm curious how you "activated all shims recommended by angular.io". Can you explain that a little bit? – Durst

So i guess this means the cdn is sufficent to resolve the issue? Well in your angular.io project there should be a file called "polyfills.ts" where all the shims from angular.io/docs/ts/latest/guide/browser-support.html are commented out. I basically removed those comments for all classes which had something to do with IE. Which would be all the core-js/es6/*, the web-animations-js and classlist.js. Hopes this helps – Amaranthine

I've been having the same problem with googlebot not being able to render my angular 4 site using "fetch as google". It returned blank no matter what I tried. The solution here with including shim, was what finally solved the problem for me. Consider making this the accepted answer. – Procopius

great answer. does the google bot also read the content of the site for SEO? – Beadsman

@Beadsman The bot is at least able to read the meta description field. If it can also follow links/routes I'm not sure since i just tested it on a small test project here modern-sub.net if you try to google "modern sub.net" you will find the same meta-description field. However i would assume that links also work since in the HTML it is a normal <a href=""> – Amaranthine

since many ppl. did see that post. At ng-conf 2018 there was a talk about SEO where the give more details: youtube.com/watch?v=fiT5g9KSxmw – Amaranthine

shouldn't webpack be transpiling to ES5 when it does the same for trypescript? Why should ES6 compatibility be an issue? – Expressage

I never claimed that it is an ES6 compatibility issue. In fact i can not completly say what the missing piece is. – Amaranthine

Okay, stop right there. Yesterday I spent a whole day removing my app from my Java deployment and made it a standalone server side rendered application in order to get indexed by google. Now I just added this line and googlebot is also able to see my website. Please somebody explain to my why this works and what exactly I am doing to my application if I add this shim.min.js thing. – Lugubrious

@StefanFalk sorry for you that you waisted so much time. It is a little bit a shame that there is no clear information existining about that. For the question about the shims. A shim just adds functionality to the JS browser functions. For example if the browser has no implementation for the string.repeat function it would just provide an own implementation on String.prototype.repeat to support "modern" code on the old browser. What they exactly changing can be seen on there sides: npmjs.com/package/core-js. FYI: i highly doubt that you need the CDN and polyfill – Amaranthine

Cool. Thanks for that additional info. By now I think I get what actually happened there. It would seem that Angular just uses new features which googlebot does not know about but loading shim.min.js corrects that for the reasons you stated. It really is a shame that there is so little information about this especially since Google should be interested in promoting Angular on one hand and making it accessible for googlebot on the other. For now I'll go that way and have a close look on what googlebot sees. The question is now: Why would anybody use Angular Universal for this instead? – Lugubrious

as " Naveed Ahmed" pointed out in the last answer. Universal doesn't fix this. But universal is still very useful since it reduces loading time and loading times influences your SEO rank.Further low end devices like older Phones might benefit from the better performance. – Amaranthine

Thanks for the solution! This answer probably saved me hours/days worth of researching.. – Engram

Answer 2

First this was talked about at NG-Conf 2018 Apr For the slides click here

Looking at the source code of the Angular.io here is how the angular guys do this according to the slides

 <script>
    if (window.document.documentMode) {
      // polyfill IE11 in a blocking way
      var s = document.createElement('script');
      s.src = 'generated/ie-polyfills.min.js';
      document.head.appendChild(s);
    } else if (!Object.assign) {
      // polyfill other non-evergreen browsers in a blocking way
      var polyfillUrl = "https://cdn.polyfill.io/v2/polyfill.min.js?features=default,Array.prototype.find&flags=gated&unknown=polyfill";
      // send a blocking XHR to fetch the polyfill
      // then append it to the document so that its eval-ed synchronously
      // this is required because the method used for IE is not reliable with other non-evergreen browsers
      var xhr = new XMLHttpRequest();
      xhr.addEventListener("load", function() {
        var s = document.createElement('script');
        s.type = 'text/javascript';
        var code = this.responseText;
        s.appendChild(document.createTextNode(code));
        document.head.appendChild(s);
      });
      xhr.open("GET", polyfillUrl, false);
      xhr.send();
    }
  </script>

Add the above script in the HEAD section of your index file.

Worth to mention that if you go with the answer of just adding a CDN you most probably are loading a script that is not needed for most of the modern browsers to those browsers as well and should be avoided.

Driggers

Answer 3

Since your frontend isn't being served by any server side language, I'd recommend using Angular2-Universal to serve a static HTML site on initial load.

You can checkout their quickstart guide and get it working pretty fast.

Fireside

Isn't Googlebot supposed to render what a normal user sees on the browser? – Sangria

@NileshG Yes, and a normal users sees exactly that, an initial loading screen. – Fireside

yea i would prefer an universal page. However it is simply not true that it is just working with that. see my answer above. – Amaranthine

Even with Universal you need to add the shims mentioned by @Amaranthine in the answer. – Teevens