How can i get my container to go from starting -> healthy

Question

Background: My Docker container has a very long startup time, and it is hard to predict when it is done. And when the health check kicks in, it first may show 'unhealthy' since the startup is sometimes not finished. This may cause a restart or container removal from our automation tools.

My specific question is if I can control my Docker container so that it shows 'starting' until the setup is ready and that the health check can somehow be started immediately after that? Or is there any other recommendation on how to handle states in a good way using health checks?

Side question: I would love to get a reference to how transitions are made and determined during container startup and health check initiating. I have tried googling how to determine Docker (container) states but I can't find any good reference.

Josuejosy

Answer 1

My specific question is if I can control my container so that it shows 'starting' until the setup is ready and that the health check can somehow be started immediately after that?

I don't think that it is possible with just K8s or Docker.
Containers are not designed to communicate with Docker Daemon or Kubernetes to tell that its internal setup is done.
If the application takes a time to setup you could play with readiness and liveness probe options of Kubernetes.

You may indeed configure readynessProbe to perform the initial check after a specific delay.
For example to specify 120 seconds as initial delay :

readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 120

Same thing for livenessProbe:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
    httpHeaders:
    - name: Custom-Header
      value: Awesome
  initialDelaySeconds: 120
  periodSeconds: 3

---

For Docker "alone" while not so much configurable you could make it to work with the --health-start-period parameter of the docker run sub command :

--health-start-period : Start period for the container to initialize before starting health-retries countdown

For example you could specify an important value such as :

docker run --health-start-period=120s ...

Roxana

Sorry. I just realized there is a big difference in k8s and docker status. And I was looking specifically for Docker status. I will update the question. Sorry for the confusion. – Josuejosy

It doesn't matter. I edited with a Docker way. But as said at the beginning you have not direct way to achieve what you want. In a general way containers are not designed to be healthy after an important delay. I think that for these corner cases we need to play with options provided to specify a start time for health check – Roxana

I suspected that but was hoping for some unknown hidden hero feature. :D – Josuejosy

...like playing around with PIDs – Josuejosy

In fact that hero feature would be not a good present because coupling strongly the application inside the container with the docker orchestrator adds much complexity. As I am not super clever I am always looking for KISS (Keep it simple, stupid) :) – Roxana

Answer 2

Here is my work around. First in docker-compose set long timeout, start_period+timeout should be grater than max expected starting time, eg.

    healthcheck:
      test: ["CMD", "python3", "appstatus.py", '500']
      interval: 60s
      timeout: 900s
      retries: 2
      start_period: 30s

and then run script which can wait (if needed) before return results. In example above it is appstatus.py. In the script is something like:

timeout = int(sys.argv[1])
t0 = time.time()
while True:
    time.sleep(2)
    if isReady():
        sys.exit(os.EX_OK)
    t = time.time() - t0
    if t > timeout:
        sys.exit(os.EX_SOFTWARE)
   

Biocellate