aggregating gradle multiproject test results using TestReport

Question

I have a project structure that looks like the below. I want to use the TestReport functionality in Gradle to aggregate all the test results to a single directory. Then I can access all the test results through a single index.html file for ALL subprojects. How can I accomplish this?

.
|--ProjectA
  |--src/test/...
  |--build
    |--reports
      |--tests
        |--index.html (testresults)
        |--..
        |--..
|--ProjectB
    |--src/test/...
      |--build
        |--reports
          |--tests
            |--index.html (testresults)
            |--..
            |--..

Dev

Answer 1

From Example 4. Creating a unit test report for subprojects in the Gradle User Guide:

subprojects {
    apply plugin: 'java'

    // Disable the test report for the individual test task
    test {
        reports.html.enabled = false
    }
}

task testReport(type: TestReport) {
    destinationDir = file("$buildDir/reports/allTests")
    // Include the results from the `test` task in all subprojects
    reportOn subprojects*.test
}

Fully working sample is available from samples/testing/testReport in the full Gradle distribution.

Bimetallism

Hey Pete, I need some direction. In my case, I have only one project i.e. ProjectA and after gradle clean build integrationTest, I see build/jacoco/test.exec and build/jacoco/integrationTest.exec. Under build/reports/tests/xxxxxx I see either index.html for Unit tests or only Integration tests i.e. if I run gradle clean build, I see unit tests index.html and if I run gradle clean build integrationTest, then it overwrites the data in build/reports/tests/xxx and gets the new data for integrationTest task in the same folder (build/reports/tests). – Ferwerda

Similarly, When I run sonar-runner, I do see both .exec files getting picked during JacocoSensor and I also see jacoco overall .exec getting created in the workspace under .sonar folder and under build/reports/jacoco folder. sonar-runner completes successfully, but I dont see the results in project's dashboard in Sonar even though I have both widgets set for showing Unit tests/coverage and Integration tests. Please advise what I may be missing. – Ferwerda

You need to set a separate report dir for integration tests (see Gradle Build Language Reference). Not sure about the Sonar problem. – Bimetallism

testReport is deprecated in Gradle 1.9 and shouldn't be used any more. – Doublebreasted

It's only the test.testReport property that's deprecated. As the deprecation message says, you just need to replace it with test.reports.html.enabled = false. – Bimetallism

I'm just saying :) also, this doesn't work unless (like the questioner asked) both projects are in subdirectories; I had my parent project in the root directory, to no avail. – Doublebreasted

The question is about aggregating the test reports of all subprojects. The answer works regardless of the directory layout. – Bimetallism

I have found that if one test fail, task report is not completed - neither the rest of the tests are executed, so is needed to add ignoreFailures to test task (source). – Anesthesiology

How I like when people pointing to /current/ url in documentation, as if this chapter be there forever. – Managerial

docs.gradle.org/5.6.2/userguide/… – Managerial

Could not get unknown property 'test' for project ':my-subproject' of type org.gradle.api.Project. – Kwa

It's better to use reportOn getTasksByName("test", true) since that'll work if some subprojects doesn't have any test tasks – Amato

Answer 2

In addition to the subprojects block and testReport task suggested by @peter-niederwieser above, I would add another line to the build below those:

tasks('test').finalizedBy(testReport)

That way if you run gradle test (or even gradle build), the testReport task will run after the subproject tests complete. Note that you have to use tasks('test') rather than just test.finalizedBy(...) because the test task doesn't exist in the root project.

Stank

I had to do the following fopr Gradle 5.6.2: rootProject.getTasksByName('test', true).each { it.finalizedBy(testReport) } – Disagreeable

Answer 3

I am posting updated answer on this topic. I am using Gradle 7.5.1.

TestReport task

In short I'm using following script to set up test aggregation form subprojects (based on @Peter's answer):

subprojects {
  apply plugin: 'java'
}

task testReport(type: TestReport) {
  destinationDir = file("$buildDir/reports/allTests")
  // Include the results from the `test` task in all subprojects
  testResults.from = subprojects*.test
}

Note that reportOn method is "deprecated" or will be soon and replaced with testResults, while at the same time testResults is still incubating as of 7.5.1. I got following warning in IDE

The TestReport.reportOn(Object...) method has been deprecated. This is scheduled to be removed in Gradle 8.0.

Hint: subproject*.test is example of star dot notation in groovy that invokes test task on a list of subprojects. Equally would be invocation of subprojects.collect{it.test}

• TestReport#reportOn (Gradle API documentation)
• TestReport#testResults (Gradle API documentation)
• reportOn replacement for gradle 8 (Gradle Forum)

test-report-aggregation plugin

There is also alternative option for aggregating tests (Since Gradle 7.4). One can apply test-report-aggregation plugin. If your projects already apply java plugin, this means they will come with jvm-test-suite, all you have to do is apply the plugin.

plugins {
  id 'test-report-aggregation'
}

Then you will be able to invoke test reports through testSuiteAggregateTestReport task. Personally didn't use the plugin, but I think it makes sense to use it if you have multiple test suites configured with jvm-test-suite.

Example project can be found in https://github.com/gradle-samples/Aggregating-test-results-using-a-standalone-utility-project-Groovy

Redevelop

The plugin seems not working on an Android project. The Test Report Aggregation plugin does not currently work with the com.android.application plugin. – Merlon

Answer 4

If using kotlin Gradle DSL

val testReport = tasks.register<TestReport>("testReport") {
    destinationDir = file("$buildDir/reports/tests/test")
    reportOn(subprojects.map { it.tasks.findByPath("test") })

subprojects {
    tasks.withType<Test> {
        useJUnitPlatform()
        finalizedBy(testReport)
        ignoreFailures = true
        testLogging {
            events("passed", "skipped", "failed")
        }
    }
}

And execute gradle testReport. Source How to generate an aggregated test report for all Gradle subprojects

Orfield

destinationDir and buildDir are deprecated – Merlon

Answer 5

For 'connectedAndroidTest's there is a approach published by google.(https://developer.android.com/studio/test/command-line.html#RunTestsDevice (Multi-module reports section))

1. Add the 'android-reporting' Plugin to your projects build.gradle.

   apply plugin: 'android-reporting'

2. Execute the android tests with additional 'mergeAndroidReports' argument. It will merge all test results of the project modules into one report.

   ./gradlew connectedAndroidTest mergeAndroidReports

Jared

Although the doc says it can also merge the unit tests, running this task depends on espresso tests which is not something I was hoping for. – Kozloski

This plugin was now [removed] from the documentation. (issuetracker.google.com/issues/222730176) – Merlon

Answer 6

FYI, I've solved this problem using the following subprojects config in my root project build.gradle file. This way no extra tasks are needed.

Note: this places each module's output in its own reports/<module_name> folder, so subproject builds don't overwrite each other's results.

subprojects {
 // Combine all build results
  java {
    reporting.baseDir = "${rootProject.buildDir.path}/reports/${project.name}"
  }
}

For a default Gradle project, this would result in a folder structure like

build/reports/module_a/tests/test/index.html
build/reports/module_b/tests/test/index.html
build/reports/module_c/tests/test/index.html

Chlodwig

I dont see how this really answer the question. The question asks about ONE index.html with accumulated results. – Ipomoea

I wanted something exactly like that but this didn't worked for my multi module android project – Decretive

Answer 7

If you're defining a custom integrationTest tasks in your subprojects, and want to aggregate their results, then give this a try in your main project build.gradle:

evaluationDependsOnChildren()

task aggregateIntegrationTestReports(type: TestReport) {
    destinationDir = file("$buildDir/reports/tests/integration")

    def integrationTestsBinaryResultsDirs = subprojects.stream().map(subproject -> subproject.tasks.findByName("integrationTest"))
            .filter(task -> task != null)
            .map((Test task) -> {
                mustRunAfter task // configure that aggregateIntegrationTestReports cannot be run before any of integrationTest
                task.finalizedBy aggregateIntegrationTestReports // configure that if you run integrationTest task, aggregateIntegrationTestReports will be run automatically after it
                return task.binaryResultsDirectory
            })
            .map(binaryResultsDirectory -> files(binaryResultsDirectory)?.singleFile)
            .toList()
    getTestResults().setFrom(integrationTestsBinaryResultsDirs) // configure report aggregation
}

Stipend