Erlang's slave module: code and IO distribution

Question

We use Erlang with the slave module in an application that is spawning slave nodes on different machines that report to and are coordinated by a master node on another machine. The slave nodes are opening ports on their machines and running some external programs (essentially, the erlang slave nodes (we call them workers) are just fancy wrappers around the external programs).

However, we encountered some unexpected problems for which we have not found good solutions.

1. Code distribution. Currently our Makefile rsyncs the compiled erlang code (the ebin folder) to the machines running the worker nodes and we load it via the -pa argument on worker node startup. There really should be some way of automatically distributing the code at runtime via Erlang but we are not sure how to do it.

2. Logging. The slave module documentation says "All TTY output produced at the slave will be sent back to the master node". However, when we run lager (the basho logger) on our slave (worker) nodes, its output does not get redirected to the master node's tty (there's only the log output from the master node). Currently we have a process running on the master node that logs (via lager) messages it gets from the slave nodes. So to log something on the slave nodes, we send messages to the master node.

We're starting the worker nodes like this:

slave:start(IP, NodeName, NodeArgs)

where NodeArgs is

 -setcookie thecookie -pa /home/dau/mapro/deps/nicedecimal/ebin/ /home/dau/mapro/deps/meck/ebin/ /home/dau/mapro/deps/lager/ebin/ /home/dau/mapro/deps/jsx/ebin/ /home/dau/mapro/apps/worker/ebin/ /home/dau/mapro/ebin/ -s worker_app

where all given paths are absolute paths on the machines running the worker nodes.

Lorolla

the slave module docs: erlang.org/doc/man/slave.html – Lorolla

These are good questions, but they need separating into separate questions to better fit the SO norms. – Caldron

Answer 1

You're doing the right thing as far as item #1: you are responsible for installation of the Erlang source modules or precompiled beam files on the remote host machines, just as you are for installation of OTP/Erlang itself. Once they are in the code path on the remote machines you can load and unload and hotswap to you heart's content, though :)

wrt item #2: have you verified that your slave module code can output naive error messages (i.e. erlang:display()) to your master node? in which case the problem is lager configuration - try setting error_logger_redirect to false as a quick test, perhaps... apologies in advance if none of this is useful!

Masticatory

wrt #1: I can use nl/0 to load a module on all nodes. Sadly, this isn't recursive so it really only loads the given module. Also, I want to start an application on the slave node and apparently the .app file has to be on that node's machine. – Lorolla

wrt #2: erlang:display() does not display anything in the master node's shell. error_logger_redirect does not seem to have any effect on this. – Lorolla

@Lorolla have you found a solution to this? – Node

@Node sadly, not really. We still use the workarounds described in the question. Please post an answer here if you find a better solution. – Lorolla