fig up: docker containers start synchronisation

Question

For one of my home projects I decided to use docker containers and fig for orchestration (first time using those tools).

Here is my fig.yaml:

rabbitmq:
  image: dockerfile/rabbitmq:latest
mongodb:
  image: mongo
app:
  build: .
  command: python /code/app/main.py
  links:
   - rabbitmq
   - mongodb
  volumes:
   - .:/code

Rabbitmq starting time is much slower than loading time of my application. Even though rabbitmq container starts loading first (since it is in app links), when my app tries to connect to rabbitmq server it's not yet available (it's definately loading timing problem, since if I just insert sleep for 5 seconds before connecting to rabbitmq - everything works fine). Is there some standard way to resolve loading time synchronisation problems?

Thanks.

Kidwell

Answer 1

I don't think there is an standard way to solve this, but it is a known problem and some people have acceptable workarounds.

There is a proposal on the Docker issue tracker about not considering a container as started until it is listening at the exposed ports. However it likely won't be accepted due to other problems it would create elsewhere. There is a fig proposal on the same topic as well.

The easy solution is to do the sleep like @jcortejoso says. An example from http://blog.chmouel.com/2014/11/04/avoiding-race-conditions-between-containers-with-docker-and-fig/:

function check_up() {
    service=$1
    host=$2
    port=$3

    max=13 # 1 minute

    counter=1
    while true;do
        python -c "import socket;s = socket.socket(socket.AF_INET, socket.SOCK_STREAM);s.connect(('$host', $port))" \
        >/dev/null 2>/dev/null && break || \
        echo "Waiting that $service on ${host}:${port} is started (sleeping for 5)"

        if [[ ${counter} == ${max} ]];then
            echo "Could not connect to ${service} after some time"
            echo "Investigate locally the logs with fig logs"
            exit 1
        fi

        sleep 5

        (( counter++ ))
    done
}

And then use check_up "DB Server" ${RABBITMQ_PORT_5672_TCP_ADDR} 5672 before starting your app server, as described in the link above.

Another option is to use docker-wait. In your fig.yml.

rabbitmq:
  image: dockerfile/rabbitmq:latest
mongodb:
  image: mongo
rabbitmqready:
  image: aanand/wait
  links:
   - rabbitmq
app:
  build: .
  command: python /code/app/main.py
  links:
   - rabbitmqready
   - mongodb
  volumes:
   - .:/code

Sessions

Answer 2

Similar problems I have encountered I have solved using a custom script set up as CMD in my Dockerfiles. Then you can run any check command you wish (sleep for a time, or waiting to the service be listening, for example). I think there is not a standard way to do this, anyway I think the best way would be the application run could be able to ask the external service to be up and running, and the connect to them, but this is not possible in most cases.

Blessed

Answer 3

For testing on our CI, we built a small utility that can be used in a Docker container to wait for linked services to be ready. It automatically finds all linked TCP services from their environment variables and repeatedly and concurrently tries to establish TCP connections until it succeeds or times out.

We also wrote a blog post describing why we built it and how we use it.

Cassette