How to build stateful docker services architecture with CoreOS's fleet?

Question

CoreOS used with fleet enables one to build services running some docker applications.

But is there any way to run docker services which require its state to be kept between restarts, to be persistent? For instance, databases or services that must store some files to be shared later on.

Because as far as I know, the service can be launched on core-1 machine (for example), and on restart will be launched on another one randomly. So the docker volume can be lost.

Ambush

Answer 1

The simplest way to maintain a database service is to always schedule the fleet unit to the same machine. You can do this by adding an [X-Fleet] section to the fleet unit file and either assigning the unit to a particular X-ConditionMachineID, or X-ConditionMachineMetadata. See the coreos documentation.

You can then persist data outside your docker containers by mounting a volume from the host machine. The recommended way of doing this is to wrap this data in a separate data container via docker:

docker run --name mongodb-volume -v /home/core/mongodb-data:/data/db busybox docker run -p 27017 --volumes-from mongodb-volume mongodb:latest

Since /home/core/mongodb-data on a particular machine will store persistent mongodb state and the unit will always be scheduled to that same machine, this will solve your problem.

Keare

But... if that machine needs to restart, you no longer have a database server running. – Jitterbug

That could be true, but is database-specific. You could be using couchbase server where the database is still running, but portions of it are unavailable while a particular node restarts. – Keare

@Keare Not true. Couchbase cluster should not constantly rebalance. Why not keep Couchbase outside of the Coreos cluster and connect the DB cluster + Coreos cluster via private ip's. Regards Hareem Haque – Impressive

Answer 2

You can consider to run some distributed filesystem over CoreOS cluster. This way whatever machine your database service container ends on, it will always be able to use database, mounted from DFS.

Niemi

Are you implying that something like zookeeper should be installed on top of CoreOS? Or something completely different? – To

No, I mean distributed filesystem like GlisterFS, Ceph and alike. – Niemi