How to make a Spring Boot application quit on tomcat failure

Question

We have a bunch of microservices based on Spring Boot 2.5.4 also including spring-kafka:2.7.6 and spring-boot-actuator:2.5.4. All the services use Tomcat as servlet container and graceful shutdown enabled. These microservices are containerized using docker.
Due to a misconfiguration, yesterday we faced a problem on one of these containers because it took a port already bound from another one.
Log states:

Stopping service [Tomcat]
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
***************************
APPLICATION FAILED TO START
***************************

Description:

Web server failed to start. Port 8080 was already in use.

However, the JVM is still running, because of the kafka consumers/streams.

I need to destroy everything or at least do a System.exit(error-code) to trigger the docker restart policy. How I could achieve this? If possible, a solution using configuration is better than a solution requiring development.

I developed a minimal test application made of the SpringBootApplicationand a KafkaConsumer class to ensure the problem isn't related to our microservices. Same result.

POM file

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.5.4</version>
  <relativePath/> <!-- lookup parent from repository -->
</parent>
...
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.kafka</groupId>
  <artifactId>spring-kafka</artifactId>
</dependency>

Kafka listener

@Component
public class KafkaConsumer {

  @KafkaListener(topics = "test", groupId = "test")
  public void process(String message) {

  }
}

application.yml

spring:
  kafka:
    bootstrap-servers: kafka:9092

Log file

2021-12-17 11:12:24.955  WARN 29067 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.PortInUseException: Port 8080 is already in use
2021-12-17 11:12:24.959  INFO 29067 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2021-12-17 11:12:24.969  INFO 29067 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2021-12-17 11:12:24.978 ERROR 29067 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   : 

***************************
APPLICATION FAILED TO START
***************************

Description:

Web server failed to start. Port 8080 was already in use.

Action:

Identify and stop the process that's listening on port 8080 or configure this application to listen on another port.

2021-12-17 11:12:25.151  WARN 29067 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-test-1, groupId=test] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE}
2021-12-17 11:12:25.154  INFO 29067 --- [ntainer#0-0-C-1] org.apache.kafka.clients.Metadata        : [Consumer clientId=consumer-test-1, groupId=test] Cluster ID: NwbnlV2vSdiYtDzgZ81TDQ
2021-12-17 11:12:25.156  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Discovered group coordinator kafka:9092 (id: 2147483636 rack: null)
2021-12-17 11:12:25.159  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] (Re-)joining group
2021-12-17 11:12:25.179  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] (Re-)joining group
2021-12-17 11:12:27.004  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Successfully joined group with generation Generation{generationId=2, memberId='consumer-test-1-c5924ab5-afc8-4720-a5d7-f8107ace3aad', protocol='range'}
2021-12-17 11:12:27.009  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Finished assignment for group at generation 2: {consumer-test-1-c5924ab5-afc8-4720-a5d7-f8107ace3aad=Assignment(partitions=[test-0])}
2021-12-17 11:12:27.021  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Successfully synced group in generation Generation{generationId=2, memberId='consumer-test-1-c5924ab5-afc8-4720-a5d7-f8107ace3aad', protocol='range'}
2021-12-17 11:12:27.022  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Notifying assignor about the new Assignment(partitions=[test-0])
2021-12-17 11:12:27.025  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Adding newly assigned partitions: test-0
2021-12-17 11:12:27.029  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Found no committed offset for partition test-0
2021-12-17 11:12:27.034  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-test-1, groupId=test] Found no committed offset for partition test-0
2021-12-17 11:12:27.040  INFO 29067 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-test-1, groupId=test] Resetting offset for partition test-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka:9092 (id: 11 rack: null)], epoch=0}}.
2021-12-17 11:12:27.045  INFO 29067 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : test: partitions assigned: [test-0]

Phosphoresce

When properly integrated this shouldn't happen. So I wonder how you have setup things as to listen to kafka, looks like you do that outside the scope/management of Spring (Boot). – Franz

Hi @M.Deinum I've updated my question including a test case of a new boot application – Phosphoresce

Answer 1

I’ve raised an issue to Spring kafka folks and it results to be a bug. No configuration is needed, spring application should quit as expected. Reference: https://github.com/spring-projects/spring-kafka/issues/2055

As stated in the issue, fix is cherry picked to spring-kafka 2.7.x, 2.6.x, 2.5.x. I’ll upgrade to the first 2.7.x patch release which includes the fix.

Phosphoresce

Answer 2

Since you have everything containerized, it's way simpler.

Just set up a small healthcheck endpoint with Spring Web which serves to see if the server is still running, something like:

@RestController(
public class HealtcheckController {

  @Get("/monitoring")
  public String getMonitoring() {
    return "200: OK";
  } 

}

and then refer to it in the HEALTHCHECK part of your Dockerfile. If the server stops, then the container will be scheduled as unhealthy and it will be restarted:

FROM ...

ENTRYPOINT ...
HEALTHCHECK localhost:8080/monitoring

If you don't want to develop anything, then you can just use any other Endpoint that you know it should successfully answer as HEALTCHECK, but I would recommend that you have one endpoint explicitly for that.

Length

Thank you for the proposed solution. Before posting I thought we could use actuator (already included) with its health endpoint, but I would like to solve it in the spring-boot application if possible – Phosphoresce