wait for other deployments to start running before other can be created?

Question

I am creating the deployments/services using REST APIs. I send POST request with bodies which contain the JSON objects which create the applications on Openshift. After I call all the APIs, these objects get instantiated.

I have 2 deployments which are dependent on mongodb deployment but this mongodb takes a little longer to start running, while the two deployments which are dependent on mongodb start running earlier. This breaks the code inside the 2 deployments as the mongodb connection fails(since it is not up yet).

There could be 2 possible way I can fix this problem.

1. I put a delay after i create mongodb deployment and recursively call the API to check it's status if it is running or not.

2. Just like we make changes in docker-compose, with the key, depends-on which tell the docker-compose that all the dependencies should be started first and then the dependent container.

Is there any way this could be achieved in openshift?

Prunelle

Answer 1

Instead of implementing complex logic for dependency handling, use health checking mechanism of Kubernetes. If your application starts and doesn't see Mongo DB, let it crash. Kubernetes will keep restarting it until Mongo DB comes online, and your application becomes healthy and serving as well. Kubernetes won't send traffic to not yet healthy instances.

Docs: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

Cranium

This approach creates unnecessary load as your pod keeps starting up and then crashing. – Hobie

Answer 2

Just like we make changes in docker-compose, with the key, depends-on which tell the docker-compose that all the dependencies should be started first and then the dependent container.

• You might want to look into Init Containers for dependent container. They run to completion before container is actually started. Below excerpt is taken from referenced documentation (given below) for use cases that might be applicable to your issue:

  • They run to completion before any app Containers start, whereas app Containers run in parallel, so Init Containers provide an easy way to block or delay the startup of app Containers until some set of preconditions are met.

    Examples

    Here are some ideas for how to use Init Containers:

    • Wait for a service to be created with a shell command like:

      for i in {1..100}; do sleep 1; if dig myservice; then exit 0; fi; done; exit 1
      

    • Register this Pod with a remote server from the downward API with a command like:

      curl -X POST http://$MANAGEMENT_SERVICE_HOST:$MANAGEMENT_SERVICE_PORT/register -d ‘instance=$()&ip=$()’
      

    • Wait for some time before starting the app Container with a command like sleep 60.

• Reference documentation:
  https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

Malinda

Answer 3

Alex has pointed out correct practice to follow with kubernetes. But if you still want directly depend on other pod phase you can use this pod-dependency-init-container that I have build. This will check if any pod with given labels is running before starting your pod.

Wikiup