Spring Config server not reachable with docker-compose until client is restarted

Question

I have a 'discovery first' setup with Eureka, Config Server, and my client.

The issue is that these 3 services start in order, but the client-server seems to register too early, and can never find config-server. I've tried a third-party library that allows a wait until config-server:8888 is available, but that doesn't always seem to work either. It's similar to a race condition.

The workaround is that if I docker restart the client-server after everything is up, it registers and finds config-server just fine.

First run of docker-compose:

Fetching config from server at : http://localhost:8888
Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available

When I docker restart the client:

Fetching config from server at : http://a80b001d04a7:8888/
Located environment: name=client-server, profiles=[default], label=null, version=053c8e1b14dc0281d5af0349c9b2cf012c1a346f, state=null

Not sure if my JAVA_OPTS properties aren't being set fast enough from my docker-compose.yml, or there is some networking race condition, or what. I've been going back and forth on this for too long.

My configuration is below:

Here's my docker-compose.yml:

version: '3'
services:
  eureka:
    image: eureka-server:latest
    environment:
    - "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
    ports:
      - 8761:8761
  config:
    image: config-server:latest
    environment:
      - "JAVA_OPTS=-DEUREKA_SERVER=http://eureka:8761/eureka"
    depends_on:
      - eureka
    ports:
      - 8888:8888
  client:
    image: client-server:latest
    environment:
      JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
    depends_on:
      - config
    ports:
      - 9000:9000

Here's the eureka-server application.yml:

server:
  port: 8761

spring:
  application:
    name: eureka-server

eureka:
  client:
    registerWithEureka: false
    fetchRegistry: false
    service-url:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

Here's the config-server bootstrap.yml:

server:
  port: 8888

eureka:
  client:
    serviceUrl:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

spring:
  application:
    name: config-server

Here's the client-server bootstrap.yml:

spring:
  application:
    name: client-server
  cloud:
    config:
      discovery:
        enabled: true
        serviceId: config-server
      fast-fail: true
    retry:
      max-attempts: 10000
      max-interval: 1000

eureka:
  instance:
    hostname: client-server
  client:
    registerWithEureka: true
    fetchRegistry: true
    serviceUrl:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

Edit:

Using the docker-compose wait library (https://github.com/ufoscout/docker-compose-wait), I can have the client-server wait for eureka and config to be available, then wait 90 seconds (Eureka documentation suggests that registration could take up to 90 seconds), and it seems to work consistently.

Is this an acceptable solution? Feels like a bit of a hack.

Ditter

what error is displayed? – Claypan

@JonathanJohx it is up in the question - there is a timeout exception to localhost:8888. In the second run (after the docker restart), it finds the config-server at a80b001d04a7:8888 – Ditter

Oh right, so you need to wait first the config server and eureka server then the others services. right? – Claypan

Yea, check out my edit above. docker-compose doesn't have any built in wait capability, so you have to add it in yourself. A 90 second wait seems to do the trick, but just feels like a hack. – Ditter

Yes, I know it, I felt as you since one doesn't know when a service is up, but I found a way to do it I think it is properly, it is including a file bash in order to check if the services are up firstly if they are up then the service starts. – Claypan

Yea, that's basically what that open source library does. Thanks for your comments! – Ditter

I think it is not the same, since you put 90 seconds, but you don't know if it is really 90 more or less, while if you create a bash file and check the services are up then the others services are up, anyway you can continue using that library, kind regards. – Claypan

Answer 1

Being purist the answer to your question is NO, it is not an acceptable solution, because as it is stated here, Docker removed healthcheck from v3 on for some reason:

Docker have made a conscious decision not to support features that wait for containers to be in a "ready" state. They argue that applications depending on other systems should be resilient to failure.

In the same link, it is described why:

The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.

To handle this, your application should attempt to re-establish a connection to the database after a failure. If the application retries the connection, it should eventually be able to connect to the database.

Basically then, there are three options:

1. Use v2.1 with healhcheck. See an example here
2. Use v3 and a tool like wait-for-it or dockerize as @ortomala-lokni already perfectly explained
3. Make your application resilient to config-server failure and able config-client to retry the connection on startup

The recommended and acceptable solution is 3). You can use Spring Retry as it is mentioned here. Find below the bootstrap.yml configuration:

spring:
  application:
    name: config-client
  profiles:
     active: dev
  cloud:
    config:
     discovery:
       enabled: true
       service-id: config-server
     fail-fast: true
     retry:
       initial-interval: 1500
       multiplier: 1.5
       max-attempts: 10000
       max-interval: 1000

eureka:
  instance:
    hostname: config-client
  client:
    registerWithEureka: true
    fetchRegistry: true
    serviceUrl:
      defaultZone: ${EUREKA_SERVER:http://localhost:8761/eureka}

BTW I found an error in your spring configuration. It is fail-fast and not fast-fail.

Remember to include the following dependencies (or similar if you are using gradle):

<dependency>
    <groupId>org.springframework.retry</groupId>
    <artifactId>spring-retry</artifactId>
</dependency>

You can find a very well configuration (and explanation) here taking also into account resiliency during the registering process in the Eureka Server.

When having a microservices environment we must think of the resiliency of our environment when platform services like config-service, discovery-service are not available for a short period of time.

But I am not a purist at all and I would not have removed some functionality people is using (it is a question of freedom). So, an alternative solution is:

If it is working for you, then go ahead

Because I do not really understand why Docker suppressed the fantastic healthcheck command from v3.

Faithfaithful

This is a really thorough answer to a question that doesn't necessarily have a single correct answer. I appreciate your time. You also caught a typo in my configs. Marking this as correct and bounty winner since this is a really good read for anyone in the same situation. – Ditter

Really appreciate your comments and glad to help! – Faithfaithful

Answer 2

The best solution is probably, as Carlos Cavero said, to make your application resilient to config-server failure. But you can also solve the problem by using the wait-for script from Eficode on Github.

Copy the script into your container and in your docker-compose.yml use:

client:
    image: client-server:latest
    environment:
      JAVA_OPTS: -DEUREKA_SERVER=http://eureka:8761/eureka
    depends_on:
      - config
    ports:
      - 9000:9000
    command: wait-for $CONFIGSERVER_SERVICE_NAME:$CONFIGSERVER_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS

The environment variables for CONFIGSERVER_SERVICE_NAME and CONFIGSERVER_PORT can be defined in your Docker Compose environment file.

If you need to wait for multiple services, you can merge this pull request and list all needed services in the command line parameters such as:

command: wait-for $SERVICE1_NAME $SERVICE1_PORT $SERVICE2_NAME $SERVICE2_PORT -- java $JVM_OPTIONS -jar client.war $SPRING_OPTIONS

Restitution

Answer 3

Service dependency are always tricky when using docker-compose.

Your solution is acceptable because "there is no other way". To avoid third-part libs, this is what I do in the same scenario:

In the Dockerfile I add netcat-openbsd, a bash file I call entrypoint and the application jar and then I run the entrypoint.sh.

FROM openjdk:8-jdk-alpine
RUN apk --no-cache add netcat-openbsd
COPY entrypoint.sh /opt/bin/
COPY app.jar /opt/lib/
RUN chmod 755 /opt/esusab-bi/bin/app/entrypoint.sh

The entrypoint file has the following instruction:

#!/bin/sh

while ! nc -z config 8888 ; do
    echo "Waiting for upcoming Config Server"
    sleep 2
done

java -jar /opt/lib/app.jar

It will delay the application start-up until your config server is up, without a specific interval.

Parfitt

Answer 4

Just a friendly tip: You should not bind Config to Eureka but the other way around -> Eureka should be Config client.

Concha