Don't fail the gradle build if a test is failing with the gradle-android-test-plugin

Question

I set my project up to run with Robolectric and the the gradle-android-test-plugin. This all works fine and first tests are running and failing.

If a test fails this will also fail the gradle build. Is there a way to just let the build go on and write down the failing tests for evaluation in a later step?

The plan is to integrate the testing in a continuous integration server and there the build should only be unstable if a test fails.

Commixture

Answer 1

The correct syntax with AndroidConnectedTests is as following:

project.gradle.taskGraph.whenReady {
    connectedAndroidTest {
        ignoreFailures = true
    }
}

Or for newer Gradle-plugin (1.3.0 and later), try:

project.gradle.taskGraph.whenReady {
    connectedDebugAndroidTest {
        ignoreFailures = true
    }
}

But if you have Flavors (in newer Gradle versions):

android {

    // ...

    project.gradle.taskGraph.whenReady {
        android.productFlavors.all { flavor ->
            // Capitalize (as Gralde is case-sensitive).
            def flavorName = flavor.name.substring(0, 1).toUpperCase() + flavor.name.substring(1)

            // At last, configure.
            "connected${flavorName}DebugAndroidTest" {
                ignoreFailures = true
            }
        }
    }
}

Now the test task is not failing the build anymore, which ensures the coverage report is created, and you can pick up the failed tests with your build server to mark the build as unstable etc.

Commixture

If you're running task (e.g.) connectedAndroidTestFoo instead of connectedAndroidTest then second line of the script (see above) should be connectedAndroidTestFoo {. Foo is a flavour – Mariner

Since 1.3.0 gradle version, you should check this answer #30101281 – Ferret

It looks like this no longer works. "gradle.startParameter.continueOnFailure = true" has a similar effect – Pitterpatter

Answer 2

Hmm. Well you have two options I think. One is to use

testTask.ignoreFailures = true

to not let the task fail when a test fails.

Another approach would be to run your gradle command with '--continue'. This will execute as many tasks as possible and list the failed tasks at the end and not stop after the first task has failed.

Generator

There is no testTask in my project to change this value on. Maybe this is specific to the Java Testing tasks? I'm using Gradle for Android. – Commixture

"testTask" is just the placeholder for your test task. you have to set the ignoreTestFailures property to true on the task of type "Test" that is failing your build – Generator

There is a task called test but setting this property is doing nothing. But I don't know if this is the correct test since the tests are executed by a third party plugin: android-test. – Commixture

Is this property suppose to be ignoreFailures = true ? – Janeyjangle

@KevinO, your right. typo on my side. it should be 'ignoreFailures'. I've updated my answer – Generator

This does not work with the gradle-android-test-plugin, does it? – Kharif

Do you mind elaborating the testTask.ignoreFailures = true sample? I.e. where one should put it? Is it in the project.gradle.taskGraph.whenReady block? – Mariner

Could not find property 'testDebug' on task – Jokester

Answer 3

late to this but you could do:

    testOptions {
        unitTests.all {
            ignoreFailures = true 
        }
    }

within android {} block

Nordstrom

Above does nothing for Android-Tests (tests running on device/ Emulator), and any failed android-test blocks build process. – Cutcheon

As a side note; 99% of our tests are usually Android-tests, which can do unit-tests as well, hence Gradle's "unitTests" name is confusing, and should be something like "javaTests" instead (as only Java API is available, and calling any Android API causes error). – Cutcheon