How to solve liquibase waiting for changelog lock problem in several pods in OpenShift cluster?

Question

We are supporting several microservices written in Java using Spring Boot and deployed in OpenShift. Some microservices communicate with databases. We often run a single microservice in multiple pods in a single deployment. When each microservice starts, it starts liquibase, which tries to update the database. The problem is that sometimes one pod fails while waiting for the changelog lock. When this happens in our production OpenShift cluster, we expect other pods to fail while restarting because of the same problem with changelog lock issue. So, in the worst case scenario, all pods will wait for the lock to be lifted.

We want liquidbase to automatically prepare our database schemas when each pod is starting.

Is it good to store this logic in every microservice? How can we automatically solve the problem when the liquidbase changelog lock problem appears? Do we need to put the database preparation logic in a separate deployment?

So maybe I should paraphrase my question. What is the best way to run db migration in term of microservice architecture? Maybe we should not use db migration in each pod? Maybe it is better to do it with separate deployment or do it with some extra Jenkins job not in OpenShift at all?

Pettis

How do you know that the liquibase has successfully updated the db? – Kamikamikaze

@ChrisBolton After launching the pod with the new version of application where we changed something in database structure via liquidbase scripts we just look into postgres and compare what is written in liquibase script and what we have in postgres. This is fully manual procedure. – Pettis

Automate the process. – Kamikamikaze

@ChrisBolton, we really don't know how to – Pettis

Hi, have you found a solution yet? We kind of run into the same problem and it is really annoying... – Argentum

@Argentum Yes, we solved it by removing liquibase migration at service start time. Now separate Jenkins job apply the migration and separate job start service and service it self doesn’t apply database update – Pettis

Answer 1

We're running liquibase migrations as an init-container in Kubernetes. The problem with running Liquibase in micro-services is that Kubernetes will terminate the pod if the readiness probe is not successful before the configured timeout. In our case this happened sometimes during large DB migrations, which could take a few minutes to complete. Kubernetes will terminate the pod, leaving DATABASECHANGELOGLOCK in a locked state. With init-containers you will not have this problem. See https://www.liquibase.org/blog/using-liquibase-in-kubernetes for a detailed explanation.

UPDATE Please take a look at this Liquibase extension, which replaces the StandardLockService, by using database locks: https://github.com/blagerweij/liquibase-sessionlock

This extension uses MySQL or Postgres user lock statements, which are automatically released when the database connection is closed (e.g. when the container is stopped unexpectedly). The only thing required to use the extension is to add a dependency to the library. Liquibase will automatically detect the improved LockService.

I'm not the author of the library, but I stumbled upon the library when I was searching for a solution. I helped the author by releasing the library to Maven central. Currently supports MySQL and PostgreSQL, but should be fairly easy to support other RDBMS.

Antilles

This looks really useful! Any chance of this getting incorporated by Liquibase itself? – Mansion

@DirkLuijk great question. I've just added the extension on the Liquibase available extensions list: liquibase.jira.com/wiki/spaces/CONTRIB/pages/1998865/… – Antilles

Also supports Oracle and MariaDB – Mortise

@Antilles in the repo's docs you state that the package com.github.blagerweij.sessionlock must be added to Liquibase classpath scanner's whitelist. How can I do this programmatically with Spring Boot and the revised Liquibase 4+ concept of Scope where they have refactored the ServiceLocator in a way that it no longer has the #addPackageToScan() method? – Dunite

@DanieleRepici Since the jar file contains a services file in META-INF, that should work out of the box, without any additional configuration. Just add a dependency to your gradle or maven build file. Please let me know if that works. – Antilles

Answer 2

When Liquibase kicks in during the spring-boot app deployment, it performs (on a very high level) the following steps:

1. lock the database (create a record in databasechangeloglock)
2. execute changeLogs;
3. remove database lock;

So if you interrupt application deployment while Liquibase is between steps 1 and 3, then your database will remain locked. So when you'll try to redeploy your app, Liquibase will fail, because it will treat your database as locked.

So you have to unlock the database before deploying the app again.

There are two options that I'm aware of:

1. Clear databasechangeloglock table or set locked to false. Which is DELETE FROM databasechangeloglock or UPDATE databasechangeloglock SET locked=0
2. Execute liquibase releaseLocks command. You can find documentation about it here and here.

Obloquy

yeah, thanks, you have described problem right. But the question is can I release lock in logic of my application and each pod in my case will try to release this lock when starts? I think this will cause concurrent problem. Am I right? How and where release this lock automatically and is it possible to do it in my SpringBoot application code? Please pay attention to the fact application is running in many pods in one deployment in OpenShift, so there are several instances of my app running at the same time – Pettis

We do it manually now, by hands, but how and where to write code for this? We want this problem to be solve automatically – Pettis

Can you execute liquibase releaseLocks command in your deployment pipeline? You're right, Liquibase locks the database to solve the problem of concurrency. Perhaps you can override some Liquibase class and add "unlocking logic" to it, so it executes before Liquibase checks for the log, or something like that. – Obloquy

So maybe I should paraphrase my question. What is the best way to run db migration in term of microservice architecture? Maybe we should not use db migration in each pod? Maybe it is better to do it with separate deployment or do it with some extra Jenkins job not in OpenShift at all? – Pettis

Oh, I see. In my experience, if multiple microservices connect to the same database, then it's reasonable to create a deployment order, so microservices deploy one by one. Also, microservice architecture at it's finest should have each microservice connect to it's own database. Otherwise it looks more like distributed monolith. – Obloquy

No the problem is a little bit different. We have one SpringBoot application, this application have liquibase script. Each application is working with exact one database, but the problem is that each application is running on multiple pods of OpenShift. So this mean that every instance of one application is trying to launch migration and the can do it concurrently. The question is how to achieve one application run liquidbase in one database if this application runs in multiple pods? – Pettis

Sorry, looks like I misunderstood your question. I'm not very familiar with OpenShift. I've just described the general idea and how to resolve the error itself. – Obloquy

Oh, sorry this is because I do not practice my English for a long time and can't properly describe our needs. Thank you for your answer, we do understand the concept of lock mechanism, but the really interesting thing is how to achieve it in OpenShift/Kubernetes. So the idea we have is to run one db update pod on separate deployment, but we are really interesting in finding some better solution. Or maybe some people faces the same problem it will be very interesting to learn someones experience – Pettis

Answer 3

We managed to solve this in my company by following also the same approach Liquibase suggests with Init Containers, but instead of using a new container and run the Liquibase migration via Liquibase CLI, we are reusing the existing Spring Boot service setup but just executing the Liquibase logic. We have created an alternative main class that can be used in an entrypoint to populate the database using Liquibase.

The InitContainerApplication class brings the minimal configuration required to start the application and set up Liquibase.

Typical usage:

entrypoint: "java -cp /app/extras/*:/app/WEB-INF/classes:/app/WEB-INF/lib/* com.backbase.buildingblocks.auxiliaryconfig.InitContainerApplication"

Here the class

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.SpringBootConfiguration;
import org.springframework.boot.autoconfigure.ImportAutoConfiguration;
import org.springframework.context.ApplicationContext;

@SpringBootConfiguration
@ImportAutoConfiguration(InitContainerAutoConfigurationSelector.class)
public class InitContainerApplication implements ApplicationRunner {

    @Autowired
    private ApplicationContext appContext;

    public static void main(String[] args) {
        SpringApplication.run(InitContainerApplication.class, args);
    }

    @Override
    public void run(ApplicationArguments args) throws Exception {
        SpringApplication.exit(appContext, () -> 0);
    }

}

Here is the use as an Init Container:

spec:
  initContainers:
    - name: init-liquibase
      command: ['java']
      args: ['-cp', '/app/extras/*:/app/WEB-INF/classes:/app/WEB-INF/lib/*',
                 'com.backbase.buildingblocks.auxiliaryconfig.InitContainerApplication']

Circumvent

How does this prevent the database from being left in a locked state ? init-container or not, if the pods gets whipped (when a new deployment is applied for example) before liquibase releases the locked who is going to unlock it? Am I missing something with init-container ? – Underdeveloped

This is just another approach to the Liquibase proposal to avoid long term migrations being killed by k8s readiness/liveness probes (liquibase.org/blog/using-liquibase-in-kubernetes for a detailed explanation) but instead of using a new container with the Liquibase CLI we reuse the same container with our service but using a different entrypoint. If you kill the pod during a migration you get the locked status and you need to solve that manually, unless you change the LockService, like in the example shown in previous answers: github.com/blagerweij/liquibase-sessionlock – Circumvent

Answer 4

Finally we solved this problem in another project by removing liquibase migration at microservice start time. Now separate Jenkins job apply the migration and separate Jenkins job deploy and start microservice after migration apply. So now microservice itself doesn’t apply database update

Pettis

Answer 5

I encountered this issue when one of the Java applications I manage abruptly shut down.

The logs were displaying the error below when the application tries to start:

waiting to acquire changelock

Here's how I solved it

I fixed this issue by:

• Stopping the application
• Deleting the databasechangelog and databasechangelog.lock files in the database connected to the application.
• Restarting the application

In my case the application was connected to 2 databases. I had to delete the databasechangelog and databasechangelog.lock files in the both databases and then restarted the application. The both database databasechangelog and databasechangelog.lock files have to be at sync.

After this the application was able to acquire changelock file.

Beiderbecke