Merging Integration and Unit test reports with JaCoCo

Question

I am generating failsafe and surefire reports with maven with the JaCoCo plugin, but I can only manage to get them in separate reports. I would like to have a overall coverage view (merge between both Unit Tests and Integration Tests) as well.

After what I think was a thorough google search I could only found a way to do this with Sonar. Is there any simpler way to do this?

Related question: Maven separate Unit Test and Integration Tests

Jereme

Answer 1

You should take a look into the documentation of JaCoCo Maven plugin which contains a merge goal.

Dashing

Good answer! Feel free to see my code example below. – Langelo

Answer 2

I recently implemented this: after some headaches and a lot of testing, I have a configuration that works beautifully.

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>${jacoco.version}</version>
    <executions>
        <execution>
            <id>pre-unit-test</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
                <propertyName>surefireArgLine</propertyName>
            </configuration>
        </execution>
        <execution>
            <id>pre-integration-test</id>
            <goals>
                <goal>prepare-agent-integration</goal>
            </goals>
            <configuration>
                <destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
                <propertyName>testArgLine</propertyName>
            </configuration>
        </execution>
        <execution>
            <id>post-integration-test</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
            </configuration>
        </execution>
        <execution>
            <id>post-unit-test</id>
            <phase>prepare-package</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
            </configuration>
        </execution>
        <execution>
            <id>merge-results</id>
            <phase>verify</phase>
            <goals>
                <goal>merge</goal>
            </goals>
            <configuration>
                <fileSets>
                    <fileSet>
                        <directory>${project.build.directory}/coverage-reports</directory>
                        <includes>
                            <include>*.exec</include>
                        </includes>
                    </fileSet>
                </fileSets>
                <destFile>${project.build.directory}/coverage-reports/aggregate.exec</destFile>
            </configuration>
        </execution>
        <execution>
            <id>post-merge-report</id>
            <phase>verify</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/coverage-reports/aggregate.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.18.1</version>
    <configuration>
        <argLine>${surefireArgLine}</argLine>
        <skipTests>${skip.unit.tests}</skipTests>
        <includes>
            <include>**/*UT.java</include>
            <include>**/*MT.java</include>
            <include>**/*Test.java</include>
        </includes>
        <skipTests>${skipUTMTs}</skipTests>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.12.4</version>
    <configuration>
        <skipTests>${skipTests}</skipTests>
        <skipITs>${skipITs}</skipITs>
        <argLine>${testArgLine}</argLine>
        <excludes>
            <exclude>**/*UT*.java</exclude>
        </excludes>
    </configuration>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

As you can see, there are 6 distinct Jacoco executions to run the tests, merge the files and create an aggregate report. On top of the Jacoco config, you also need to configure Surefire and Failsafe to take an argument from Jacoco (Surefire runs the unit tests and Failsafe runs the integration tests).

All of the configuration that I used should be there, what you do with it is your design architecture to make it fit within your desired requirements. Personally, I recommend a look into what I exclude and include within surefire and failsafe if you are having issues with files not being read.

Langelo

I would recommend not to change the default naming conventions of maven-surefire-plugin and the maven-failsafe-plugin cause it is not necessary. Keep the conventions...And I would strongly recommend to use the same versions for maven-surefire and maven-failsafe furthermore the most recent versions... – Dashing

Probably a worthy recommendation. I simply wanted to give users a code example that works instead of "Hey, go check 'this' out". – Langelo

@BobVanDeHey thanks a lot for this snippet. It was very useful for me to setup my JaCoCo configurations. – Teratogenic

Always enjoy helping the community! Let me know if you want more clarification. – Langelo

@ChadVanDeHey In your configuration for surefire skipTests seems set twice. – Bourgogne

Thanks. I had headaches as well until I found this :D – Bratcher

Thank you for this post! I was able to make mine work following your note. – Plus

Do we put this code at the root/parent POM? The module POMs have their own jacoco-maven-plugin settings, I just need to put this in the parent? – Benco

way too complicated , can be done much simpler with way less lines, see my solution below – Okra

Answer 3

Not the answer you're looking for, but still...

In most cases, you should not merge coverage from unit and integration tests.

The value of unit tests is that they improve the design of your application and ensure that corner cases of your code is working correctly. You should try to have a high branch coverage of your unit tests.

The value of your integration tests is that they ensure that the main use cases of your application are working correctly and that the whole stack is integrated correctly. You should try to have a high functional coverage for your integration tests. (But it is fairly hard to measure functional coverage with a tool).

If you need integration tests to improve your branch coverage, that's a strong indication that you should review the design of your code. And if you already have a high branch coverage without integration tests, adding them should not modify significantly your metrics.

Intrigue

Some classes in JPA package can't be unit-tested because they don't expose the fields for asserting. – Sherylsheryle

Answer 4

Building on top of the answer of Chad, here is my configuration. It is configured as a separate profile named jacoco so I can turn it on and off easily. It uses only the default configuration. It can create a separate code coverage report for unit tests, separate code coverage report for integration tests and also a combined code coverage report.

    <profile>
        <id>jacoco</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.jacoco</groupId>
                    <artifactId>jacoco-maven-plugin</artifactId>
                    <executions>
                        <execution>
                            <id>pre-unit-test</id>
                            <goals>
                                <goal>prepare-agent</goal>
                            </goals>
                        </execution>
                        <execution>
                            <id>post-unit-test</id>
                            <phase>test</phase>
                            <goals>
                                <goal>report</goal>
                            </goals>
                        </execution>
                        <execution>
                            <id>pre-integration-test</id>
                            <goals>
                                <goal>prepare-agent-integration</goal>
                            </goals>
                        </execution>
                        <execution>
                            <id>post-integration-test</id>
                            <goals>
                                <goal>report-integration</goal>
                            </goals>
                        </execution>
                        <execution>
                            <id>merge-results</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>merge</goal>
                            </goals>
                            <configuration>
                                <fileSets>
                                    <fileSet>
                                        <directory>${project.build.directory}</directory>
                                        <includes>
                                            <include>*.exec</include>
                                        </includes>
                                        <excludes>
                                            <exclude>aggregate.exec</exclude>
                                        </excludes>
                                    </fileSet>
                                </fileSets>
                                <destFile>${project.build.directory}/aggregate.exec</destFile>
                            </configuration>
                        </execution>
                        <execution>
                            <id>post-merge-report</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>report</goal>
                            </goals>
                            <configuration>
                                <dataFile>${project.build.directory}/aggregate.exec</dataFile>
                                <outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>

Eisenstark

Answer 5

Maybe more nice way is to achieve is use same jacoco file, but let it another tests, which works for us - unit and it tests.

            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.1</version>
                <executions>
                    <execution>
                        <id>default-prepare-agent</id>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                        <configuration>
                            <skip>${maven.surefire.skipTests}</skip>
                            <propertyName>maven.surefire.argLine</propertyName>
                            <!-- using the same dest file for both UT and IT -->
                            <destFile>${sonar.jacoco.reportPath}</destFile>
                        </configuration>
                    </execution>
                    <execution>
                        <id>default-prepare-agent-integration</id>
                        <goals>
                            <goal>prepare-agent-integration</goal>
                        </goals>
                        <configuration>
                            <skip>${maven.failsafe.skipTests}</skip>
                            <propertyName>maven.failsafe.argLine</propertyName>
                            <!-- append to the UT dest file -->
                            <destFile>${sonar.jacoco.reportPath}</destFile>
                            <append>true</append>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <skipTests>${maven.surefire.skipTests}</skipTests>
                <failIfNoTests>${maven.surefire.failIfNoTests}</failIfNoTests>
                <!-- allow argLine to be modified by other plugins, e.g. jacoco -->
                <argLine>@{maven.surefire.argLine}</argLine>
            </configuration>
        </plugin>
        <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <configuration>
                <skipTests>${maven.failsafe.skipTests}</skipTests>
                <failIfNoTests>${maven.failsafe.failIfNoTests}</failIfNoTests>
                <!-- allow argLine to be modified by other plugins, e.g. jacoco -->
                <argLine>@{maven.failsafe.argLine}</argLine>
            </configuration>
        </plugin>

Supersede

Answer 6

This works out-of-the-box.

Explanations

The prepare-agent goal integrates by default with both SureFire and FailSafe plugins (to be exact the argLine parameter set by prepare-agent is consumed by both plugins). Additionally, prepare-agent by default configures Jacoco so that it appends coverage results if the file already exists, so in the end you get single target/jacoco.exec file which contains results from both unit test and integration tests, combined.

Just in case this is the relevant config, as you see nothing changed from default:) :

    <plugins>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.22.2</version>
        <executions>
          <execution>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>


      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.22.2</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>


      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.8.6</version>
        <executions>
          <execution>
            <goals>
              <goal>prepare-agent</goal>
              <goal>report</goal>
              <goal>check</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>

Chattel

If it doesn't work out-of-the-box for someone, the problem could be that an explicit <argLine> parameter is set for the plugin. In this case, it may be necessary to add the JaCoCo property that holds the parameter for specifying the agent to the explicitly specified <argLine>, as it's done in some of the other solutions here. – Heres

Answer 7

I get the point where Unit tests should really be at the source of code coverage... but sometimes, one would like to have the ability to know right? Here is what I did (this is from using gradle).

plugins {
    [...]
    id 'jacoco'
}

[...]

test {
    jacoco { /* specify the "exec" data file name (see the "Unit" in there) */
        destinationFile = file("$buildDir/jacoco/jacocoTestUnit.exec")
        classDumpDir = file("$buildDir/jacoco/classpathdumpsUnit")
    }
    useJUnitPlatform { /* I use the JUnit Jupiter @Tag Annotation to create my suites... you are free to do something else */
        excludeTags 'IT'
    }
    description = "Run unit tests"
    group = "verification"
}

task intTest(type: Test) {
    jacoco {
        destinationFile = file("$buildDir/jacoco/jacocoTestInt.exec")
        classDumpDir = file("$buildDir/jacoco/classpathdumpsInt")
    }
    useJUnitPlatform {
        includeTags 'IT'
    }
    description = "Run integration tests"
    group = "verification"
}

jacocoTestReport {
    /*
     * find all test exec files and feed them to jacoco
     */
    def list = []
    def dir = new File("${buildDir}/jacoco/")
    if(dir.exists()) {
        dir.eachFileRecurse(FileType.FILES) { file ->
            /* gather all the "exec" files available */
            if (file.getName().startsWith("jacocoTest") && file.getName().endsWith(".exec")) {
                list << file.getAbsolutePath()
            }
        }
        /* provide all the "exec" files to jacoco */
        executionData.from = files(list)
    }

    /* 
     * you must run all tests before running jacoco.
     * We want the liberty to run partial tests instead of all,
     * so this task doesn't depend on any test task.
     */
    reports {
        xml.enabled true
    }
}

With this, you can have the the possibility to get the coverage from Unit tests with:

./gradlew clean test jacocoTestReport

Or you can have the coverage from Integration tests with:

./gradlew clean intTest jacocoTestReport

Or you can have the overall coverage of both unit and Integration tests with:

./gradlew clean test inTest jacocoTestReport

Disclaimer: I'm no Jacoco or Gradle expert... feel free to comment anything I may have forgotten. This worked beautifully so far for my needs.

Academe

I think this idea is legit and very similar to the concept from the Gradle documentation: docs.gradle.org/6.4-rc-1/samples/… (still works with more recent versions of Gradle). – Silveira

Answer 8

In order to merge the reports here, I have a complete working solution.

Please note that in order to work properly the merge strategy, the phases should be executed sequentially (if the mvn test and mvn verify -DskipUnitTests will be executed in parallel there is possible to not work fine).

<!-- Jacoco is used to generate the reports for SonarQube -->
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.6</version>

    <configuration>
        <skip>${skipTests}</skip>
    </configuration>

    <executions>
        <!-- Prepares the property pointing to the JaCoCo runtime agent which
             is passed as VM argument when Maven the Surefire plugin is executed. -->
        <execution>
            <id>pre-unit-test</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
                
                <!-- Sets the name of the property containing the settings for JaCoCo runtime agent. -->
                <propertyName>surefireArgLine</propertyName>
            </configuration>
        </execution>
        
        <!-- Make sure that after the Unit Tests execution, the jacoco-ut.exec file is generated,
             will be merged to the aggregation file -->
        <execution>
            <id>post-unit-merge</id>
            <phase>test</phase>
            <goals>
                <goal>merge</goal>
            </goals>
            <configuration>
                <fileSets>
                    <fileSet>
                        <directory>${project.build.directory}</directory>
                        <includes>
                            <include>**/coverage-reports/*.exec</include>
                        </includes>
                    </fileSet>
                </fileSets>
                <destFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</destFile>
            </configuration>
        </execution>
        
        <!-- Ensures that the code coverage report is created/updated after unit tests have been run. -->
        <execution>
            <id>post-unit-test</id>
            <phase>test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <dataFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</dataFile>
                
                <!-- Sets the output directory for the code coverage report. -->
                <outputDirectory>${project.reporting.outputDirectory}/jacoco</outputDirectory>
            </configuration>
        </execution>

        <!-- Prepares the property pointing to the JaCoCo runtime agent which
            is passed as VM argument when Maven the Failsafe plugin is executed. -->
        <execution>
            <id>pre-integration-test</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
                
                <!-- Sets the name of the property containing the settings for JaCoCo runtime agent. -->
                <propertyName>failsafeArgLine</propertyName>
            </configuration>
        </execution>
        
        <!-- Make sure that after the Integration Test execution, the jacoco-it.exec file is generated,
             will be merged to the aggregation file -->
        <execution>
            <id>post-integration-merge</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>merge</goal>
            </goals>
            <configuration>
                <fileSets>
                    <fileSet>
                        <directory>${project.build.directory}</directory>
                        <includes>
                            <include>**/coverage-reports/*.exec</include>
                        </includes>
                    </fileSet>
                </fileSets>
                <destFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</destFile>
            </configuration>
        </execution>
        
        <!-- Ensures that the code coverage report is created/updated after integration tests have been run. -->
        <execution>
            <id>post-integration-test</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <dataFile>${project.build.directory}/coverage-reports/jacoco_aggregation.exec</dataFile>
                
                <!-- Sets the output directory for the code coverage report. -->
                <outputDirectory>${project.reporting.outputDirectory}/jacoco</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <!-- Sets the VM argument line used when unit tests are run. -->
        <argLine>${surefireArgLine}</argLine>
    </configuration>
</plugin>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <configuration>
        <!-- Sets the VM argument line used when integration tests are run. -->
        <argLine>${failsafeArgLine}</argLine>
    </configuration>
</plugin>

Now, since the reports are generated, can be executed the sonar command in order to publis the report:

mvn sonar:sonar -Dsonar.coverage.jacoco.xmlReportPaths="target/site/jacoco/jacoco.xml"

Tophole

tried it 18.03.2024 and it works perfectly. thx. – Dory

Answer 9

Below configuration works well and generates an aggregated code coverage report comprising code covered by unit and integration tests.

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.10</version>
    <executions>
        <execution>
            <id>pre-unit-test</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <append>true</append>
                <destFile>${project.build.directory}/jacoco-ut.exec</destFile>
            </configuration>
        </execution>
        <execution>
            <id>post-unit-test</id>
            <phase>test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/jacoco-ut.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
            </configuration>
        </execution>
        <execution>
            <id>pre-integration-test</id>
            <goals>
                <goal>prepare-agent-integration</goal>
            </goals>
            <configuration>
                <append>true</append>
                <destFile>${project.build.directory}/jacoco-it.exec</destFile>
            </configuration>
        </execution>
        <execution>
            <id>post-integration-test</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/jacoco-it.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
            </configuration>
        </execution>
        <execution>
            <id>merge-all-results</id>
            <phase>verify</phase>
            <goals>
                <goal>merge</goal>
            </goals>
            <configuration>
                <fileSets>
                    <fileSet>
                        <directory>${project.build.directory}</directory>
                        <includes>
                            <include>jacoco*.exec</include>
                        </includes>
                    </fileSet>
                </fileSets>
                <destFile>${project.build.directory}/jacoco-aggregate.exec</destFile>
            </configuration>
        </execution>
        <execution>
            <id>merged-report</id>
            <phase>verify</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/jacoco-aggregate.exec</dataFile>
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-aggregate</outputDirectory>
            </configuration>
        </execution>
        <execution>
            <id>code-coverage-report-check</id>
            <goals>
                <goal>check</goal>
            </goals>
            <configuration>
                <dataFile>${project.build.directory}/jacoco-aggregate.exec</dataFile>
                <rules>
                    <rule>
                        <element>BUNDLE</element>
                        <limits>
                            <limit>
                                <counter>COMPLEXITY</counter>
                                <value>COVEREDRATIO</value>
                                <minimum>0.9</minimum>
                            </limit>
                        </limits>
                    </rule>
                </rules>
            </configuration>
        </execution>
    </executions>
</plugin>

Careycarfare

Answer 10

all answers above a way too complicated an totaly unnecessary, because:

The only thing you have to make sure is, that the report is generated after the integration-test phase. Because by default the Jacoco report is run before the integration-test phase.

Surprisingly enough, this:

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.10</version>
    <executions>
        <execution>
            <id>jacoco-prepare-agent</id>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <execution>
            <id>jacoco-report</id>
            <!-- needed in order to make the report also include the results from the integration tests -->
            <phase>post-integration-test</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
    </executions>
</plugin>

already adds the additional necessary command line parameters for coverage measuring not only to the unit-tests but also to the integration-tests. But since the report is generated by default before integration-tests their coverage is not included in the test by default.

So just adding this:

<execution>
    <id>jacoco-report</id>
    <!-- needed in order to make the report also include the results from the integration tests -->
    <phase>post-integration-test</phase>
    <goals>
        <goal>report</goal>
    </goals>
</execution>

is already sufficient to add the integration-tests coverage to the report.

All the claims on the internet, and in chatGPT for a much more complicated configuration using Jacocos merge goal or defining <argLine>${argLine}</argLine> for surefire-plugin or <argLine>${failsafe.argLine}</argLine> for the fail-safe-plugin is not necessary at all!

Okra

this does create both jacoco.exec and jacoco-it.exec, but the csv/xml/html reports in site/jacoco only include the unit-tests and not the integration tests. Thus the integration with, for example, jenkins does not work this way. – Syncarpous