End to end integration test for multiple spring boot applications under Maven

Question

What is the recommended way of running an end to end integration test for multiple Spring boot applications in the Maven build's verify phase?

Basically, I have a multi-module Maven project where several modules are separate spring boot applications. These separate applications have their own configuration for data sources, integration flows with JMS queues, etc. For example, application A will poll a database for an event, and when that occurs, it produces a JSON file of data and puts a message on a JMS queue. Application B is polling the JMS queue, so picks up the message, reads the file, does some processing using another database, and puts a message on a different queue. Application C will then pick up that message, etc, etc.

I have set up integration tests for the individual applications; these run under the Maven failsafe plugin. However, I would like to integration test the whole system, end to end, under Maven. I have set up a separate module in the project dedicated to this task, and so would like the verify build phase of this module to do the end to end testing using the other dependent modules.

Is there a best practice way of doing this? I see 3 potential ways:

1. Load each application's configuration into the same application context. However, because of multiple data sources etc, this creates conflicts, and so these data sources would all have to be manually configured just to enable end to end integration testing - so this seems wrong to me.
2. Launch each application as a separate process - how then to properly keep track of them and make sure they get shut down if the test module build stops/crashes/etc?
3. Is there a way to easily load separate spring boot applications, each with its own configuration context, in the same process? This would seem to be the most sensible option. Are there any considerations in respect of the Maven build/failsafe plugin?

Mamelon

process-exec-maven-plugin could be a good option to perform the integration testing of multiple spring-boot application: You may consider my tested solution for that https://mcmap.net/q/586060/-integration-testing-spring-boot-based-microservices – Elul

Answer 1

Very nice question! I would by myself be interested what other people answer. I'll share my opinion.

In my understanding first of all you should know what exactly you want to test. Integration tests should work with an application of at least with a part of it and ensure that the component you've developed works properly in a semi-real environment. It seems like you've already done that.

Now, regarding system tests (I intentionally differentiate between integration and system tests). These should 'mimic' QA guys :) So, they treat a system as a black box. They can't invoke any internal APIs and run real flows. End-to-end tests IMO fall into this category.

In this case you would like to check them against the system deployed like in production, with the classpath like in production.

So I don't really believe in option 1 just like you.

Regarding the option 3 I'm not sure whether its a good solution as well. Even if you run your stuff with different application contexts (I don't know much Spring boot so I can't technically comment on it), in my understanding they will share the same classpath in runtime, so probably you're in risk to get clash among your thirdparties (although I know that spring boot defines a lot of versions of jars by itself, you know what I mean) especially when you upgrade only one module and probably change the dependencies. So you don't really know what exactly runs in memory when you run follow this approach.

So, for end-to-end tests, I would go with option 2. Regarding the synchronization, probably the option would be implementing some logic at the application level in conjunction with process state tracking at the level of operating system. One more point I would like to comment on is that end-to-end tests in general are still functional testing (they check the functional behavior of the system) so in general you shouldn't check system crashes there in each test. If you check the system crash for each flow, these tests will be too slow. Of course you can maintain one relatively small test suite to check corner cases as such.

Hope this helps

Brubaker

Answer 2

Just to follow-up and say what I ended up doing (which is continuing to work well):

• I created the following Maven profiles in my testing module: "default" to skip tests by default (we use jgitflow plugin so only want end-to-end tests run when explicitly requested), "standalone-e2e" for end-to-end tests not requiring external resources such as databases (aimed at developers wanting to do a full end-to-end test), and "integrated-e2e" for end-to-end tests using real databases etc (which can get triggered as part of CI). Spring profiles (activated by the corresponding Maven profile) control the configuration of the individual components.
• For standalone-e2e, relevant plugins such as activemq-maven-plugin, hsqldb-maven-plugin etc. launch (and later shut down) resources as part of the end-to-end test, running on ports reserved with build-helper-maven-plugin. The process-exec-maven-plugin is used to launch all the components to be tested in the pre-integration-test phase (as standard Spring Boot apps), and it automatically takes care of shutting them down in the post-integration-test phase. Spring configuration and specific Maven test dependencies take care of other resources such as a fake FTP server. After all resources and components are running, the test code itself then populates the database and file system as required and triggers flows (and waits for corresponding replies etc) using JMS.
• The integrated-e2e profile is almost identical but uses "real" external resources (in our case, Amazon SQS queues, MySQL database, etc) configured in the associated Spring properties.
• All files needed for and generated by the tests (e.g. data files, HSQLDB files, log files, etc) are created under the "target" build directory, so it's easy to inspect this area to see what happened during the test, and also allow "mvn clean" to clear out everything.

I hope that's useful - it was certainly refreshing to find that whatever I needed to do, a Maven plugin existed to take care of it!

Mamelon

Thanks Gary Sedgwick for mentioning the process-exec-maven-plugin which have used to implement an integration test of the interaction of two spring-boot applications: https://mcmap.net/q/586060/-integration-testing-spring-boot-based-microservices – Elul

Gary, I am trying to do something very similar, except with a little added twist: I'm trying to use the Citrus Framework for integration testing. I've been able to set up integration tests for my services, with "mocks" for other services the SUT uses. Now I'm trying to set up an end-to-end test where all of the services are real. – Teriteria

I looked at the information provided by kmarabet regarding the usage of the process-exec-maven-plugin (and other maven plugins) and am trying to see if I can make it work for me. I have also created a separate maven module in my project whose sole purpose is to contain and execute integration tests. I'm still in the early stages of figuring out what I need to do to set the module up to run the way I need it and ran into an issue that kmarabet seems to have gotten around. – Teriteria

When I run the maven build, the IT module fails because the spring-boot-maven-plugin wants to do a "repackage" and cannot find the "main" class. Well, there is no main class. Or maybe I should say, there could be many main test classes. Not sure what to do. – Teriteria

Could you provide a brief description, with examples of your POM file(s) and project structure, of how you solved your integration/system test issue? – Teriteria

Answer 3

For certain reasons, I decided to use the second approach and had a similar problem to solve. I came up with a combination of maven-dependency-plugin and process-exec-maven-plugin.

So assume we have some applications (e.g. app_A, app_B, ...) and a module T where the tests are implemented.

The idea was to copy applications to local build directory and start them, before starting tests. After test run the applications will be stopped.

To achieve this I added in module Ts pom the following build-configurations

<build>
  ...
  <plugin>
    <!-- copy apps to current build-dir -->
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>3.1.2</version>
    <executions>
      <execution>
        <id>copy-app-A</id>
        <goals>
          <goal>copy</goal>
        </goals>
        <phase><!-- either pre-test or pre-integration-test --></phase>
        <configuration>
          <artifactItems>
            <artifactItem>
              <groupId>${project.groupId}</groupId>
              <artifactId>app_a</artifactId>
              <version>${project.version}</version>
              <outputDirectory>${project.build.directory}</outputDirectory>
              <destFileName>app_a.jar</destFileName>
            </artifactItem>
          </artifactItems>
        </configuration>
      </execution>
      <execution><!-- repeat as many apps you have --></execution>
    </executions>
  </plugin>
  <plugin>
    <!-- start applications -->
    <groupId>com.bazaarvoice.maven.plugins</groupId>
    <artifactId>process-exec-maven-plugin</artifactId>
    <version>0.9</version>
    <executions>
      <execution>
        <!-- repeat 'start' block for each copied app -->
        <id>start-external-server</id>
        <phase><!-- same as in copy block --></phase>
        <goals>
          <goal>start</goal>
        </goals>
        <configuration>
          <name>run-app-A</name>
          <arguments>
            <argument>java</argument>
            <argument>-jar</argument>
            <argument>${project.build.directory}/app_a.jar</argument>
          </arguments>
        </configuration>
      </execution>
      <execution>
        <!-- this is always executed and ensures apps get stopped -->
        <id>stop-server</id>
        <phase><!-- either post-test or post-integration-test depending on --></phase>
        <goals>
          <goal>stop-all</goal>
        </goals>
      </execution>
    </executions>
  </plugin>
  ...
</build>

Gilmore

Generally, links to a tool or library should be accompanied by usage notes, a specific explanation of how the linked resource is applicable to the problem, or some sample code, or if possible all of the above. – Contextual