Introduction to Erlang/OTP production applications deployment

Question

I would like to develop and deploy an Erlang/OTP application into production on a VPS.

I am pretty familiar with developing Erlang code on a local machine and my question is about deployment.

Basically, I would like to know what steps I should take in order to move Erlang code from a local machine to a production server and make it run, i.e. be available for users.

Note: I have read some documentation about Erlang and command line, Erlang code module, Erlang releases, but I am still not sure how to pursue the required task.

However, I guess that deploying an Erlang-based software on a server is a bit more tricky than doing sudo tasksel for LAMP.

I plan to have an Erlang/OTP application which has Mochiweb, CouchDB (couchbeam) and boss_db as dependencies.

So, my newbie questions about deploying all that stuff on a production server are the following:

• I plan to use Ubuntu Server 12.04; is there any better choice for a Linux distro to use for Erlang/OTP in production?
• How all the code should be organized? Should I put my application into a /home/myapp/ dir and then put all the dependencies into /home/myapp/deps? Or should I put all dependencies into /usr/local/lib/erlang/lib? (returned by code:get_path()). Should I somehow update the dependencies regularly or should I freeze them?
• How do I make the whole application start once the server starts? Should it be some kind of bash script or anything else?
• I know that Erlang allows hot code upgrades, but how should I organize that? On Rails I could update the code with git, does anything similar exist in the Erlang world?

Falgout

Answer 1

There are two types of dependencies: Internal and External. If you want to do it the right way(tm), it takes a bit of time getting to work:

External dependencies:

Taking the latter first, an external dependency is some other thing that has to run before your application can run. For instance a PostgreSQL database, or a Riak cluster. For those, you usually just use the usual stuff in Ubuntu for making it start up properly. I've had good experience with using monit for these tasks:

http://mmonit.com/monit/

Internal Dependencies:

For internal dependencies, you need to arrange your program into applications inside the Erlang VM. These have dependencies on each other, like the external dependencies. Your main application may need a logger running before it should start, for instance. Then you create a release. A release copies the Erlang binaries and necessary libraries/beams/applications into a release directory, forming a self-contained Erlang system. It contains a boot-script which tells how to start up the applications in the right order and keep them running. So you can tar-ball up this release, copy it to the server and then start it. There are some basics covered here:

http://learnyousomeerlang.com/release-is-the-word

but do also read the chapters before it on applications. You can also get rebar to call reltool for you to build a release. This is what I usually do.

Hot upgrades:

Handling hot upgrades in production can be done in a couple of ways. You can move the beam to the machine and then deploy it, take the shell and then call l(Module) to load it into the running system. This works for smaller fixes. For large systematic upgrades you can do a release-upgrade which will upgrade the running system on the fly without stopping service. But if your system is mostly shared nothing, it is usually not worth it. Instead, you can have multiple machines and upgrade them in sequence.

For instance, you can upgrade a machine and then use a system like HAProxy to send 2% of all requests to the new system. Then systematically turn up the request load weight.

Yellowknife

I assume that in the real world nobody pushes the source code onto production machines to update the system, but just beam files, right? – Falgout

And by "self-contained Erlang system" you mean a release with .beam files that are supposed to be executed by a production machine which has Erlang VM installed? – Falgout

Answer 2

While @I GIVE CRAP ANSWERS gave a pretty thorough summary, I feel compelled to throw in the use of sync, which helps to automate the hot-recompiling and reloading of modules.

The simple way is you specify sync as a rebar dependency, then when you're getting ready to deploy an upgrade, you can run sync:go() on the Erlang node. This starts the sync engine, which watches for filesystem changes. Then you can use git to push to your server. Sync will notice the files change, recompile them, and load the new beams automatically.

Then, you can run sync:stop() right away to tell the system to stop watching for filesystem changes (it's generally not recommended to keep sync running on a live server, just to prevent accidental recompiling if, for whatever reason, a source file changes and it's unintentional.

Tours

Although this might work in smaller things, does this not have the potential to really mess things up when there are lots of applications with messy dependencies? (i.e. what app-ups solve?) – Polyvalent

It's definitely a simpler solution for smaller upgrades (and "simple" has it's drawbacks - primarily that it's not a comprehensive solution). Major upgrades with new apps, and new-to-be-upgraded apps, yeah, you'll want to use something more comprehensive. But for pushing changes to production that don't require adding or upgrading dependencies, it's very convenient and requires very little messing about: just push the code and it's upgraded. – Tours

That said, sync will upgrade rebar deps as well (basically, anything that's loaded in the erlang system gets monitored, along with the directories of the source and beam files). So doing something like rebar update-deps will also hotload the new apps. Of course, if the app upgrade fails to compile, then you're left in an inconsistent state (with half of the dependecy app upgraded), which is obviously not a good thing. So it can work for that, but it's not the best option with a messy dependency hierarchy. – Tours

Looks interesting then; I'll take another look at it - I've been using mochiweb's reloader.erl but will checkout Sync - would be nice to replace all the appups everywhere for the simpler projects... – Polyvalent

chops, thank you for the reference to sync. I have accepted the other answer, because it is a bit more descriptive and general. Anyway, thank you. – Falgout