How to create a self-updating Node.js application?

Question

I'd like to create an application with Node.js that periodically checks for updates and installs them if there are any.

The basic components are clear to me:

• A web server (or an FTP server, a file system, ...) which contains the update packages
• A version system (such as SemVer) so that you can tell which package is newer
• A public-key algorithm for signing update packages

Then, there may be different strategies on when to check for updates and install updates:

• On application start
• On application end
• When idleing

The application may even be shut down hard and restarted automatically.

But there are some questions left:

• Is there any module on npm available that already provides such a system?
• How to organize the different versions in the file system? Basically you'd have a host and multiple (versioned) cores. Should there be one data folder for all of them?
• How to deal with npm install & co. for newly downloaded packages?
• How to deal with broken updates?
• How could you implement such a thing so that it's compatible to Heroku & co. where you don't have a permanent drive that you can drive to?

In general: How would you try to achieve a system like this?

Lineberry

Of course this would be my responsibility to stay backwards-compatible ... but the question remains: How would you do something like that? – Lineberry

@GoloRoden What solution did you end up going with ? I have a similar requirement. Tx – Servomechanical

Unfortunately I did not end up with a solution yet, we are still investigating for ideas ... :-( – Lineberry

@GoloRoden and now? It's been 5 years since your question :-) – Murton

Haha, same situation as before ;-) – Lineberry

Answer 1

First, you could use NPM itself for package and app delivery. Second, you could add a script file to add a cron job in the package.json file (e.g. 'postinstall' or 'update') for a periodic update process. In this script you could check all you need like broken updates. In Heroku you can use a Scheduler Worker for update processing.

Rincon

Answer 2

A web server would be unnecessary. Git is a prefect solution for hosting public and private code & includes version control. For private repositories you can make use of the personal access tokens.

I have created a module for automatically updating node.js applications from git repositories. It compares the local package.json with the one from your repo, then automatically clones the repo and install dependencies. You can configure your app to check for updates on application start, end, or schedule routine checks.

Auto Git Update - https://github.com/chegele/AutoGitUpdate

import AutoGitUpdate from 'auto-git-update';

const config = {
    repository: 'https://github.com/chegele/BackupPurger'
    tempLocation: 'C:/Users/scheg/Desktop/tmp/',
    ignoreFiles: ['util/config.js'],
    executeOnComplete: 'C:\\Users\\scheg\\Desktop\\worksapce\\AutoGitUpdate\\startTest.bat',
    exitOnComplete: true
}

const updater = new AutoGitUpdate(config);

updater.autoUpdate();

Sidewinder

Answer 3

So what you are saying is that you need to update your deployment with version control that can revert if an update breaks things. It also must be able to use secure sources to update AND it must be compatible with Heroku & co?

Phew, that's asking for a lot.

Sounds like you want something like Docker. https://www.docker.com/

Using it with Heroku: https://devcenter.heroku.com/articles/container-registry-and-runtime

So what is Docker? Well, briefly it's like github, but for deployments. Instead of version control on your source code, it's version control on the entire environment. So you can have your script run within an Ubuntu environment on windows. It can use a specific version of node.js. And then use a specific set of NPM dependencies already installed. And after all those layers are built, it can use a specific version of your application within that environment. You have full control over everything that is contained in the deploy image and it runs on all the OS's that docker runs on (which includes heroku). (Note that you cannot use a Windows environment except on another Windows machine with Docker on it)

So, with Docker, rather than having your script run npm updates, YOU run npm install on the development build and push to your Docker image after testing. Your deployment server can then run a pull on your Docker image to update the application, all dependencies, even OS updates, to ensure it runs as expected. If an update breaks the install though, then you can revert to using an earlier Docker image easy peasy, just like you can remove a commit from github, you can remove the pull to the image and then run it as though the update never happened. You can even spawn multiple Docker instances with different environments and versions if you wanted by creating new image builds.

Ok, so that takes care of your deployment version control. Phew!

But what about automatically checking for updates and detecting broken installs and such?

You still need to build an updater script to get the kind of behavior you want You can't expect an updater tool to know how to debug your install and determine if it is "broken" or not. Maybe your scripts are designed to throw errors. How could a third-party updater script understand your application well enough to make the determination that a rollback is necessary?

So, your updater script should manage the Docker image pulls and reverts. It can run post-pull tests prior to starting the application. You should also ensure that your application handles errors if you want it to be self-healing. That is, it is up to your application to determine if it's necessary to perform a rollback or not. Docker makes it possible (with it's version control system) for your main app to communicate to your updater that a rollback is needed. Your updater script then performs a rollback to the last known good Docker image after your main app exits (and perhaps not perform a pull again till the next version release).

Just to give a few examples use cases:

• 1. Safe Updating:

You create a Docker image with your your program in it. You have it run in an Ubuntu 16.04 environment. It contains all the needed NPM installs and Node 8. You upload to a Docker repository server. You then deploy the image to another machine and run it there. Your application is now running out of a Docker container on another machine.

Later on, you update the NPM packages on the dev build and release a new image to your Docker repository online. Your production server detects that new build, shuts down the application, performs a pull to update it's local Docker image, and starts the instance of the application again with it's local Docker image. Your application starts crashing due to some new version of a required NPM module breaking everything. It handles the errors and after 20 errors are returned, it signals to the updater script to perform a rollback and exits. Since your application shut down, the Docker container shuts down. The updater script now removes the last Docker pull from the local image and respawns an instance of your application image again with the reverted image. Any sort of damage caused to the data files in the image is gone. Maybe the updater also writes the bad version number to a JSON file, so it can keep track of what version NOT to perform a pull on. Your application is now running happy as a clam.

But not content with having a broken version out there and wanting your other packages updated, you troubleshoot the error and revert the NPM package that was causing problems on the dev build. You push the changes to your online Docker repository and increment the version number. Your application detects the new docker build, check to ensure it is not a "forbidden build" and then signals the updater to run a pull, exiting gracefully. The Docker instance ends, and the updater performs a new pull. This time around, all is well and working fine, so no need for a rollback.

• 2. Using multiple versions

But now let's say later on, you are really hammering away at your application and you've added all kinds of great functionality to it. It is now 5000% more capable but also 5000% slower. IF you had previously released a "version 1" of the application that was simple, but fast, and it does what you need it to do for some deployment on another server, then it's incredibly easy to just spawn an instance of that exact version from your Docker repository. If you need the later version, spawn that instead. Or maybe spawn a rainbow of versions on 20 different machines all over the place. No pre-setups necessary, no installing dependencies like node or NPM packages. No need to build things from source code or check to ensure that specific tools needed are compatible with this version of MacOSX or windows or some flavor of linux. No need to see if some new version of a module breaks an earlier version of your application. The app will always run in an Ubuntu 16.04 environment with all the required software that worked for it at the time. And if someone really wants to update certain aspects of the environment, they can just create a fork.

Nevertheless

Answer 4

If you are using PM2 which is very popular method scaling and running apps nowadays with node.js. Then you might want to look into the tools they are offering. Such as: https://github.com/keymetrics/pm2-auto-pull

Ernst