No tests found when using custom runner

Question

I'm using androidx.test libraries(which I migrated to recently) in my project and using custom AndroidJUnitRunner. Before migration it was all working fine but now I'm getting this error -

Started running tests Test running failed: Instrumentation run failed due to 'Process crashed.' Empty test suite.

The custom runner class which I use extends from androidx.test.runner.AndroidJUnitRunner

In my app build.gradle file I have the following setup -

testInstrumentationRunner "com.example.CustomTestRunner"

with dependencies -

androidTestImplementation "androidx.test.ext:junit:1.1.0" androidTestImplementation 'androidx.test:runner:1.1.1' androidTestImplementation 'androidx.test:core:1.1.0' androidTestImplementation "androidx.test:rules:1.1.1"

All my test classes have @RunWith(androidx.test.ext.junit.runners.AndroidJUnit4.class)

I'm stuck at this. Any help would be appreciated. Thanks.

Vladi

Answer 1

I saw this while testing using Android 4.4 When I switched to Android 6 (SDK 23) the problem went away.

I used androidx.test.ext.junit.runners.AndroidJUnit4 for my instrumented tests @RunWith(AndroidJunit4.class)

but my testInstrumentationRunner uses the package:

testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

Mixing the two different packages seems strange, but it works.

my app/build.gradle has:

android {
    defaultConfig {
        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
    }
    testOptions { 
        execution 'ANDROIDX_TEST_ORCHESTRATOR'
        unitTests {
            includeAndroidResources = true
        }
    }

    useLibrary 'android.test.runner'
    useLibrary 'android.test.base'
    useLibrary 'android.test.mock'
}

dependencies {
    //--------------------------------------------------------------------------------
    // Test Dependencies

    // Required -- JUnit 4 framework for standard unit tests.
    testImplementation "junit:junit:$rootProject.ext.junitVersion"

    androidTestImplementation "junit:junit:$rootProject.ext.junitVersion"
    androidTestImplementation "org.hamcrest:hamcrest-library:$rootProject.ext.hamcrestVersion"

    // Mockito framework for NMEA-parser unit tests.
    testImplementation "org.mockito:mockito-core:$rootProject.ext.mockitoVersion"

    // Room testing
    androidTestImplementation "androidx.room:room-testing:$rootProject.ext.roomVersion"

    // Core library
    androidTestImplementation "androidx.test:core:$rootProject.ext.testCoreVersion"
    androidTestImplementation "androidx.arch.core:core-testing:$rootProject.ext.coreVersion"

    // AndroidJUnitRunner and JUnit Rules
    // deprecated
    androidTestImplementation "androidx.test:runner:$rootProject.ext.runnerVersion"
    androidTestImplementation "androidx.test:rules:$rootProject.ext.rulesVersion"

    // Assertions
    androidTestImplementation "androidx.test.ext:junit:$rootProject.ext.junitRunnerVersion"
    androidTestUtil "androidx.test:orchestrator:$rootProject.ext.orchestratorVersion"

//    androidTestImplementation "androidx.test.ext:truth:$rootProject.ext.xTruthVersion"
//    androidTestImplementation "com.google.truth:truth:$rootProject.ext.truthVersion"

    // Espresso dependencies
//    androidTestImplementation "androidx.test.espresso:espresso-core:$rootProject.ext.espressoVersion"
//    androidTestImplementation "androidx.test.espresso:espresso-contrib:$rootProject.ext.espressoVersion"
//    androidTestImplementation "androidx.test.espresso:espresso-intents:$rootProject.ext.espressoVersion"
//    androidTestImplementation "androidx.test.espresso:espresso-accessibility:$rootProject.ext.espressoVersion"
//    androidTestImplementation "androidx.test.espresso:espresso-web:$rootProject.ext.espressoVersion"
//    androidTestImplementation "androidx.test.espresso.idling:idling-concurrent:$rootProject.ext.espressoVersion"
//    androidTestImplementation "androidx.test.espresso:espresso-idling-resource:$rootProject.ext.espressoVersion"
}

configurations {
    all {
        resolutionStrategy {
            force "androidx.recyclerview:recyclerview:$rootProject.ext.recyclerviewVersion"
            force "org.checkerframework:checker-qual:$rootProject.ext.checkerQualVersion"
            force "org.checkerframework:checker-compat-qual:$rootProject.ext.checkerQualVersion"
            force "com.google.errorprone:error_prone_annotations:$rootProject.ext.errorProneAnnotationsVersion"
        }
    }
}

and I have these library versions:

        // Core Test library
        testCoreVersion = '1.1.0'
        coreVersion = '2.0.0-alpha1'

        // Automated Test Libraries
        // Instrumentation Test Runner
        junitRunnerVersion = '1.1.0'
        runnerVersion = '1.1.1'
        rulesVersion = '1.1.1'
        xTruthVersion = '1.0.0'
        truthVersion = '0.42'
        espressoVersion = '3.1.0'
        hamcrestVersion = '1.3'
        orchestratorVersion = '1.1.0'

        // JUnit library version
        junitVersion = '4.12'

        // Mockito version
        mockitoVersion = '2.21.0'

        // Force testing dependencies
        recyclerviewVersion = '1.0.0'
        checkerQualVersion = '2.5.3'
        errorProneAnnotationsVersion = '2.3.1'

Interpellate

What exactly is the solution here? – Jimjimdandy

When I was having trouble with the error: Started running tests Test running failed: Instrumentation run failed due to 'Process crashed.' Empty test suite. I found that the problem was due to library version conflicts in my build.gradle - by using the configuration above I was able to get rid of the error and solve issues with running unit tests and instrumented tests. – Interpellate

Answer 2

I have faced the same issue when using the custom Runner class to use a custom Application class.

What was wrong:

My problem was I was passing the only the name of the class, not the package + name.

Wrong:

package com.teebalhoor.oredoh

import android.app.Application
import android.content.Context
import androidx.test.runner.AndroidJUnitRunner

/**
 * Created by Muhammad Maqsood on 13/09/2020.
 */
class CustomTestRunner : AndroidJUnitRunner() {

override fun newApplication(
    cl: ClassLoader?,
    className: String?,
    context: Context?
): Application {

    return super.newApplication(cl, OredohTestApplication::class.java.simpleName, context)
}
}

Right way:

package com.teebalhoor.oredoh

import android.app.Application
import android.content.Context
import androidx.test.runner.AndroidJUnitRunner

/**
 * Created by Muhammad Maqsood on 13/09/2020.
 */
class CustomTestRunner : AndroidJUnitRunner() {

override fun newApplication(
    cl: ClassLoader?,
    className: String?,
    context: Context?
): Application {

    return super.newApplication(cl, OredohTestApplication::class.java.canonicalName, context)
}
}

Worked fine when I replaced (Give Only Name)

OredohTestApplication::class.java.simpleName

with (Give Package + Name)

OredohTestApplication::class.java.canonicalName 

Willock

Didn't make a difference to my setup - still says "Starting 0 tests on ..." – Britisher

Answer 3

One of the possible reasons is the old Test Orchestrator (orchestrator-1.1.1.apk) or Test Services (test-services-1.1.1.apk) applications that were built for android support library components and still installed on the target device. Open Settings -> All apps, search for them and delete. When you run tests again from Android Studio new apps that are built for AndroidX will be installed and your problem may go away.

Chadchadabe

Answer 4

What helped us figure out the issue we had in our setup was running the tests in debug mode instead.

By doing that, the stack trace was preserved and we could see that our Dagger2 fake/test graph was not initialising correctly and the app was crashing before any tests could run.

I hope this helps someone running into the same issue.

Ernaernald