Introduction to Erlang/OTP continuous deployment

Question

Note: this is an evolutionary continuation from my previous question on a similar topic.

I have been searching for the 'best practices' regarding deploying and updating an Erlang/OTP release (a set of applications) for some time now, but I could not find any direct solutions descriptions, but some 'related information' only:

http://blog.equanimity.nl/blog/2013/06/04/continuous-integration-for-erlang-with-travis-ci/

https://www.youtube.com/watch?v=G0eBDWigORY

https://www.youtube.com/watch?v=0ZGHzI9F5YE

What I mean by a 'direct solution' is the answer to the following question:

Given a production Erlang/OTP cluster with an Erlang/OTP release running on several Erlang nodes that serve as a highly available (24/7) REST API, how one can regularly push new code to that production cluster? Is there any best practice for doing that which is as easy as git push heroku master with Heroku? If not, what is the simplest way to continuously re-deploy and Erlang/OTP software in production?

I have read the 'Erlang/OTP in Action' book to understand how to deal with Erlang/OTP applications and releases and it seems to me that a simple software upgrade is not that easy. When using reltool you have to generate an archive, move that archive to the production machine (scp?), unpack it there, run the Erlang shell and load the new modules into the Erlang VM.

Also, the 'Learn You Some Erlang' book states the following:

"...if you can avoid the whole procedure (which will be called relup from now on) and do simple rolling upgrades by restarting VMs and booting new applications, I would recommend you do so."

and

"It is said that divisions of Ericsson that do use relups spend as much time testing them as they do testing their applications themselves."

Additionally, here is another recommendation to avoid hot swapping for releases for Erlang in production.

If that is true, then I do not see any usefulness of 'Erlang hot code upgrades', since I would have to restart the VM with each upgrade. And this leaves me without any knowledge of a robust and tested method of regular deployment of new code for Erlang/OTP in production.

P.S. A few notes on my software requirements.

• I am writing a set of API entry points without any sessions, so there is no need to maintain session during software updates.
• There is no need to distribute the Erlang/OTP release, I am working with an in-house custom product.

Whitewood

Answer 1

Erlang hot code upgrades are useful in two cases:

1. When you have single critical node, that can't be stopped. This might be the case, when you are running Erlang on switch or router, that is critical for you infrastructure or when you upgrade software on drone, while it is still flying.

2. When you really want to preserve the state of running node (and do some alterations). It is usually easier to design your system in such a way, that you don't need to do that, than dealing with complicated upgrades.

It is better avoid hot code upgrades, because it is hard, but sometimes it IS necessary. At those times, you will put the effort to writing and testing the upgrade. And you will glad, that it can be done at all!

In your case, you have a cluster of nodes without state, so the easiest way to do an upgrade is to remove the node from cluster, upload new release on server, start the node and add it back to the cluster. If you configured application failovers between nodes, this should be easy to script using scp. It is also good practice to test new version of code first on 5% of machines and than for example 20% before you upgrade all the nodes.

There are situations, where you can copy the beam file, go to Erlang console and type:

code:purge(my_module), code:load_file(my_module).

This is not recommended for every day work, but when you do development on remote machine, it might come in handy. For example, when you are looking for lock contentions, they may appear, when you have 16 or more cores. You could create entire release every time and upload it, but it would slow the development cycle. This use of hot code upgrade might be a little bit hacky, but it is very convenient.

Khoury

thanks for explaining the cases when the hot upgrades are really needed! Could you please elaborate on "...upload new release on server, start the node..."? How is this done in the industry? What tools are used to automate the upload and start of the new release? Also, did you mean any Erlang feature when you mentioned 'application failovers between nodes'? – Whitewood

I've added link in my answer to the feature i meant, when writing about application failovers. For making releases easier, you can use rebar, relx or erlang.mk (which uses relx internally). Then we usually use plain old scp to copy release to server (only wrap it with bash scrips). There is a really cool commercial tool called Wombat, that is used mainly for monitoring, but has the orchestration feature. You upload new release via web interface and it can deploy it to specified ips or cloud providers (again: "deploy" means, stop running node, upload and unpack new release, start the node back) – Khoury

So, essentially the answer to my question is "Generate release with rebar > Stop Erlang/OTP node which is a part of an Erlang cluster > scp the release to the node > start node again > join it to the cluster" – Whitewood