How to handle new version deployement for Docker webpack application that use code splitting?

Question

After deploying a new version of my application in Docker,

I see my console having the following error that break my application:

Uncaught SyntaxError: Unexpected token '<'

In this screenshot, the source that is missing is called: 10.bbfbcd9d.chunk.js, the content of this file looks like:

(this.webpackJsonp=this.webpackJsonp||[]).push([[10],{1062:function(e,t,n){"use strict";var r=n(182);n.d(t,"a",(function(){return r.a}))},1063:function(e,t,n){var ...{source:Z[De],resizeMode:"cover",style:[Y.fixed,{zIndex:-1}]})))}))}}]);
//# sourceMappingURL=10.859374a0.chunk.js.map

This error happens because :

1. On every release, we build a new Docker image that only include chunks from the latest version
2. Some clients are running an outdated version and the server won't have a resolution for an old chunk because of (1)

Chunks are .js file that are produced by webpack, see code splitting for more information

Reloading the application will update the version to latest, but it still breaks the app for all users that use an outdated version.

A possible fix I have tried consisted of refreshing the application. If the requested chunk was missing on the server, I was sending a reload signal if the request for a .js file ended up in the wildcard route.

Wild card is serving the index.html of the web application, this for delegating routing to client-side routing in case of an user refreshing it's page

// Handles any requests that don't match the ones above
app.get('*', (req, res) => {
  // prevent old version to download a missing old chunk and force application reload
  if (req.url.slice(-3) === '.js') {
    return res.send(`window.location.reload(true)`);
  }
  return res.sendFile(join(__dirname, '../web-build/index.html'));
});

This appeared to be a bad fix especially on Google Chrome for Android, I have seen my app being refreshed in an infinite loop. (And yes, that is also an ugly fix!)

Since it's not a reliable solution for my end users, I am looking for another way to reload the application if the user client is outdated.

My web application is build using webpack, it's exactly as if it was a create-react-app application, the distributed build directory is containing many .js chunks files.

These are some possible fix I got offered on webpack issue tracker, some were offered by the webpack creator itself:

• Don't remove old builds. <= I am building a Docker image so this is a bit challenging
• catch import() errors and reload. You can also do it globally by patching __webpack_load_chunk__ somewhere. <= I don't get that patch or where to use import(), I am not myself producing those chunks and it's just a production feature
• let the server send window.location.reload(true) for not existing js files, but this is a really weird hack. <= it makes my application reload in loop on chrome android
• Do not send HTML for .js requests, even if they don't exist, this only leads to weird errors <= that is not fixing my problem

Related issues

• https://github.com/webpack/webpack/issues/10792
• https://github.com/webpack/webpack/issues/7502

How can I implement a solution that would prevent this error?

Gastrulation

this seems to be a client-side error. can you provide the relevant client-side code as well? – Hefty

hum, in my eyes this is way too less information. The syntax error disappears after refreshing the site? Does this also happen when you clear your browsers cache before loading the site the first time? – Hefty

I have updated the question, title and added as much information as possible. I have not added a reproduction, I don't see how it would help and it's not possible. The issue title could also be: How to handle webpack application served in docker for outdated clients – Gastrulation

Answer 1

If I understood the problem correctly then there are several approaches to this problem and I will list them from the simplest one to more complicated:

Use previous version to build new version from

This is by far the simplest approach which only requires to change base image for you new version.

Consider the following Dockerfile to build version2 of the application:

FROM version1

RUN ...

Then build it with:

docker build -t version2 .

This approach, however, has a problem - all old chunks will be accumulating in newer images. It may or may not desirable, but something to take into consideration.

Another problem is that you can't update you base image easily.

Use multistage builds

Multistage builds allow you to run multiple stages and include results from each stage into your final image. Each stage may use different Docker images with different tools, e.g. GCC to compile some native library, but you don't really need GCC in your final image.

In order to make it work with multi-stage build you would need to be able to create the very first image. Let's consider the following Dockerfile which does exactly that:

FROM alpine

RUN mkdir -p /app/latest && touch /app/latest/$(cat /proc/sys/kernel/random/uuid).chunk.js

It creates a new new Docker image with a new chunk with random name and put's it into directory named latest - this is important with proposed approach!

In order to create subsequent versions, we would need a Dockerfile.next which looks like this:

FROM version2 AS previous
RUN rm -rf /app/previous && mv /app/latest/ /app/previous

FROM alpine

COPY --from=previous /app /app
RUN mkdir -p /app/latest && touch /app/latest/$(cat /proc/sys/kernel/random/uuid).chunk.js

In a first stage it rotates version by removing previous version, and moving latest into previous.

During the second stage, it copies all versions there are left in the first stage, creates a new version and puts it into latest.

Here's how to use it:

docker build -t image:1 -f Dockerfile .

>> /app/latest/99cfc0e6-3773-40a0-82d4-8c8643cc243b.chunk.js

docker build -t image:2 --build-arg PREVIOUS_VERSION=1 -f Dockerfile.next .

>> /app/previous/99cfc0e6-3773-40a0-82d4-8c8643cc243b.chunk.js
>> /app/latest/2adf34c3-c50c-446b-9e85-29fb32011463.chunk.js

docker build -t image:3 --build-arg PREVIOUS_VERSION=2 -f Dockerfile.next 

>> /app/previous/2adf34c3-c50c-446b-9e85-29fb32011463.chunk.js
>> /app/latest/2e1f8aea-36bb-4b9a-ba48-db88c175cd6b.chunk.js

docker build -t image:4 --build-arg PREVIOUS_VERSION=3 -f Dockerfile.next 

>> /app/previous/2e1f8aea-36bb-4b9a-ba48-db88c175cd6b.chunk.js
>> /app/latest/851dbbf2-1126-4a44-a734-d5e20ce05d86.chunk.js

Note how chunks are moved from latest to previous.

This solution requires your server to be able to discover static files in different directories, but that might complicate local development, thought this logic might be conditional based on environment.

Alternatively you could copy all files into a single directory when container starts. This can be done in ENTRYPOINT script in Docker itself or in your server code - it's completely up to you, depends on what is more convenient.

Also this example looks at only one version back, but it can be scaled to multiple versions by a more complicated rotation script. For example to keep 3 last versions you could do something like this:

RUN rm -rf /app/version-0; \
    [ -d /app/version-1 ] && mv /app/version-1 /app/version-0; \
    [ -d /app/version-2 ] && mv /app/version-2 /app/version-1; \
    mv /app/latest /app/version-2; 

Or it can be parameterized using Docker ARG with the number of versions to keep.

You can read more about multi-stage builds in the official documentation.

Nonobservance

Thanks a lot for those solution I have not thinked of. For the first one, it seems like a good solution but as you said, if the base image needs some improvement, it will also bring some other problem so I am not considering this solution as a possible final fix. The second approach is somehow more compliant but I don't get what happen when you sweet from previous to node. . – Gastrulation

I don't get how the other solutions would help. So far, I have decided to store the build/static/js directory on git, and keep a trace of my old chunks. It does not sounds like an elegent solution. I wonder which one is the best betwwen your second option and mine. Anyway, thanks a lot for this solution, I'll still wait untill some other approach are offered – Gastrulation

Can you clarify what is not clear, so I could try to improve the answer? The idea is that you have to copy the contents of previous version into new image and then overwrite it with new version. The FROM version1 AS previous clause makes previous version available during the build so you can copy all of it, or parts of it into your new version. – Nonobservance

Alternatively, if it's applicable in you case, you can run your back-end behind CDN, which would solve this issue as well. – Nonobservance

It's not yet confirmed that it will be served behind a CDN yet. Regarding your first question, I don't know how to read Jib in a way that would help to create a jib approach. – Gastrulation

Jib on its own will not help, I just listed it as an example for what kind of things are possible in general. It's possible to create Docker image layers manually: it's just a JSON + tar archive. In your case you could have a layer per version. Then in order to create Docker image which includes say 5 last versions, you would find corresponding layers somehow and include them into a new Docker image. That's not an easy task, but it's possible. – Nonobservance

But I think multi-stage builds might work for you if you find a way to copy only last N versions from the previous build. Maybe you could put each version into a separate directory. For example latest could always go into latest and older versions will be put into a directory named after a version. Then it's much easier to select which versions to include. Your server would have to be smart to be able to look for static assets in different directories or alternatively all assets could be copied to a single location when container starts. – Nonobservance

I've updated my answer with an example of how it can be done with multi-stage builds, and removed part about other solutions, since it's a more complicated approach and might be an overkill if it can be solved differently. – Nonobservance

Thanks, the update in the multistage approach looks somehow more complicated because you now need to have two Dockerfile. I will keep the bounty till the last day. I am still waiting for an approach which is not Docker. Somehow, there must be a way to tell the user client to refresh the app without performing an additional request to the server to check if we have the latest version. The failing request or the client could detect it, I don't know how, but the failure could be used for that. That said, I like your answer. – Gastrulation

Well, if you want to completely avoid Docker then there are several thing to check: make sure index.html is never cached - this way users should never be able to download old version, which references files which don't exist. Take a look at Service Workers - it allows to intercept network requests, including the requests for resources, so you can take some custom action in case it's not available anymore. – Nonobservance

Looking at the code snippets you provided, I think you could start with adding necessary headers to prevent caching index.html and see if that helps. – Nonobservance

If you are familiar with those, where would you add that? I want to avoid to deploy and not understand the fix. You need to know that the application is deployed behind multiple proxies. – Gastrulation

I would add headers to the response which serves index.html . Take a look at this article for more details and suggestions as a starting point. Having a monitoring for the number of requests resulting in different HTTP statuses would also help you understand whether this solution helped or not, but it might not be visible right away, since many clients might already have index.html in their cache. – Nonobservance

Proxies should respect cache control headers. But it also depends were TLS connection is terminated - before proxy, or after proxy (e.g. by your service). If TLS is it's terminated after proxy, then proxy can't make any changes to the request/response because it's encrypted. If TLS is terminated before proxy, then the behavior depends on proxy settings and you would have to figure that out. – Nonobservance

Answer 2

A simple solution is that DISABLE caching of index.html

Cache-Control: no-store

Paxton

res.header('Cache-Control', 'no-store').sendFile(...) I guess that you're not very familiar with express – Paxton

Answer 3

An approach we've used in production is to have two different environments serving your .js assets. First We have the bleeding edge one: this one only knows of the most recently built version. All requests are directed towards this environment.

When a request hits eg the assets folder and no .js file can be found, we issue a redirect to the "rescue" environment. This is a simple AWS Cloudfront distribution, which is backed by an AWS S3 bucket. Upon the build of the bleeding edge environment, we push all new assets to that S3 bucket.

In case a user is using the most recent version of the application they'll comfortably be using the bleeding edge only. Once the application updates server-side, or the user was not using the latest version, all assets are served through the "backup domain". As the bleeding edge issues a redirect* instead of serving a 404, the user does not experience a problem here (apart from having to redo the request to a different location).

This setup ensures that even very old clients can continue to function. We have seen cases where Googlebot could still request assets of over 1000 deployment ago!

Big big downside: pruning of the S3 bucket is much more work. As storage is relatively cheap, for now we just keep the assets on there. As we add a chunk identifier to the file name, storage usage does not increase that much.

Something to consider is the implementation of the redirect. You'll want your application to be agnostic about its construction. We have done it in the following way:

1. Request comes in to https://example.com/assets/asset-that-is-no-longer-available.js.
2. The server detects the request was aimed at the assets directory, but the file is not there.
3. The server replaces the hostname in the request url with assets.example.com and redirects to that location.
4. The browsers asset request is redirected to https://assets.example.com/assets/asset-that-is-no-longer-available.js which is available.
5. The application continues normally.

This keeps your main Docker image free of very infrequently accessed files, and ensures your internal deploys can continue at a higher speed. It also removes the requirement that your CI should always be able to access every single previously completed deployment code.

We have been using this approach in a setup which uses Docker as well for deployments, and have not seen issues with it for any client.

Florin

Excellent answer. Thank you ! – Gastrulation