some Robolectric tests fail when run all together but pass individually

Question

I'm on Android Studio 1.2, Robolectric 3.0-rc2.

I have two test classes, one called MotdTest with one test method that tests for POJO json serialization & deserialization. The other is called UserInfoTest, which contains 4 test methods that tests for user info that I set into SharedPreferences. If I run UserInfoTest individually, all 4 test methods always pass. However, if I run all the tests, the test in MotdTest succeeds, but two methods of UserInfoTest always fails. I'm running from command line right now by calling ./gradlew test

Does anyone know why some of my tests are failing when I run all tests? In my UserInfoTest I do properly have an @After annotated method where I do cleanup by calling clear().commit() on the SharedPreferences.Editor.

UserInfoTest:

testOnSignIn() fails at assertThat(6, equalTo(prefs.getAll().size())); because the size of prefs is 0.

And testIsSignedIn() fails at assertThat(UserInfo.isSignedIn(), is(false));

@RunWith(MyRoboRunner.class)
@Config(constants = BuildConfig.class)
public class UserInfoTest {

    private String mExpectedId;

    private String mExpectedName;

    private String mExpectedEmail;

    private String mExpectedToken;

    private String mExpectedKey;

    @Before
    public void setUp() throws Exception {
        ShadowLog.stream = System.out;

        mExpectedId = "someiD";
        mExpectedName = "johnny boy";
        mExpectedEmail = "[email protected]";
        mExpectedToken = "Session Token";
        mExpectedKey = "Session Key";
    }

    @After
    public void tearDown() {
        SharedPreferences prefs = RuntimeEnvironment.application.getSharedPreferences(
                UserInfo.PREFERENCES, Context.MODE_PRIVATE);
        prefs.edit().clear().commit();
        mExpectedId = null;
        mExpectedName = null;
        mExpectedEmail = null;
        mExpectedToken = null;
        mExpectedKey = null;
    }

    @Test
    public void testOnSignIn() {        
        SharedPreferences prefs = RuntimeEnvironment.application.getSharedPreferences(
                UserInfo.PREFERENCES, Context.MODE_PRIVATE);

        UserInfo.onSignIn(mExpectedId, mExpectedName, mExpectedEmail, mExpectedKey, mExpectedToken);
        assertThat(mExpectedKey, equalTo(UserInfo.getSessionKey()));

        assertThat(mExpectedToken, equalTo(UserInfo.getSessionToken()));
        assertThat(mExpectedId, equalTo(UserInfo.getUserId()));
        assertThat(mExpectedEmail, equalTo(UserInfo.getUserEmail()));
        assertThat(mExpectedName, equalTo(UserInfo.getUserName()));

        assertThat(6, equalTo(prefs.getAll().size()));
    }

    @Test
    public void testOnSignOut() {
        UserInfo.onSignIn(mExpectedId, mExpectedName, mExpectedEmail, mExpectedKey, mExpectedToken);
        // Set Over21 to make sure we unset this value on Signout
        UserInfo.setIsOver21(true);

        UserInfo.onSignOut();
        SharedPreferences prefs = RuntimeEnvironment.application.getSharedPreferences(
                UserInfo.PREFERENCES, Context.MODE_PRIVATE);
        assertThat(UserInfo.getSessionKey(), nullValue());
        assertThat(UserInfo.getSessionToken(), nullValue());
        assertThat(UserInfo.isOver21Set(), is(false));
        assertThat(prefs.getAll().size(), equalTo(0));
    }

    @Test
    public void testIsSignedIn() {
        assertThat(UserInfo.isSignedIn(), is(false));

        UserInfo.onSignIn(mExpectedId, mExpectedName, mExpectedEmail, mExpectedKey, mExpectedToken);
        assertThat(UserInfo.isSignedIn(), is(true));

        UserInfo.onSignOut();
        assertThat(UserInfo.isSignedIn(), is(false));
    }

    @Test
    public void testIsOver21Set() {
        assertThat(UserInfo.isOver21Set(), is(false));
        UserInfo.setIsOver21(false);
        assertThat(UserInfo.isOver21Set(), is(true));
    }
}

UserInfo:

App is an Application subclass and App.getInstance() is a singleton.

private static final SharedPreferences PREFS = App.getInstance()
        .getSharedPreferences(PREFERENCES, Context.MODE_PRIVATE);

public static void onSignIn(String userId, String fullName, String email, String sessionKey,
        String sessionToken) {
    SharedPreferences.Editor editor = PREFS.edit();
    editor.putString(PROPERTY_USER_ID, userId);
    editor.putString(PROPERTY_USER_NAME, fullName);
    editor.putString(PROPERTY_USER_EMAIL, email);
    editor.putString(PROPERTY_SESSION_KEY, sessionKey);
    editor.putString(PROPERTY_SESSION_TOKEN, sessionToken);
    editor.putStringSet(PROPERTY_PENDING_SCANS, new HashSet<String>());

    editor.commit();
}
public static boolean isSignedIn() {
    return getSessionToken() != null;
}

MyRoboRunner: A modified version of this because I'm on mac, and also because I'm targeting API 22 in my project, but robolectric doesn't support up to that yet, so I run my tests against API 21.

public class MyRoboRunner extends RobolectricGradleTestRunner {

    public MyRoboRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    protected AndroidManifest getAppManifest(Config config) {
        AndroidManifest appManifest = super.getAppManifest(config);
        String moduleRoot = getModuleRootPath(config);

        //can use this line instead dynamic path resolution when AS bug is fix, or use @Config
        //FsFile androidManifestFile = appManifest.getAndroidManifestFile();
        FsFile androidManifestFile = FileFsFile.from(moduleRoot,
                appManifest.getAndroidManifestFile().getPath()
                        .replace("bundles", "manifests/full"));
        FsFile resDirectory = FileFsFile.from(moduleRoot, appManifest.getResDirectory().getPath());
        FsFile assetsDirectory = FileFsFile
                .from(moduleRoot, appManifest.getAssetsDirectory().getPath());
        return new AndroidManifest(androidManifestFile, resDirectory, assetsDirectory) {
            @Override
            public int getTargetSdkVersion() {
                //lollipop bc it's highest that robolectric 3.0 supports
                return Build.VERSION_CODES.LOLLIPOP;
            }

            @Override
            public int getMinSdkVersion() {
                return Build.VERSION_CODES.ICE_CREAM_SANDWICH_MR1;
            }
        };
    }

    private String getModuleRootPath(Config config) {
        String moduleRoot = config.constants().getResource("").toString().replace("file:", "");
        return moduleRoot.substring(0, moduleRoot.indexOf("/build"));
    }
}

Solutrean

What kind of failure do you see? Could you share the code for tests? – Isagogics

@EugenMartynov I've included some of the code I use for the tests. – Solutrean

Answer 1

You keep static references to SharedPreferences in the PREFS. I think your tests will be fixed as soon as you reset it in your tearDown method or remove static references at all. If you remove static reference then SharedPreferences file will be cleared by Robolectric itself - no need to clear it again intearDown.

Another point to mention - I'm also using Mac and I don't have any issue with RobolectricGradleTestRunnner. Sometimes I have to run clean before running the tests but nothing else.

And another thing that you mentioned robobolectric-gradle-plugin. You don't need it with Android Studio v1.1+ and android gradle plugin v1.1+

Isagogics

Answer 2

Using the robolectric gradle plugin and setting this property has all my tests passing:

robolectric {
    // Specify max number of processes (default is 1)
    maxParallelForks = 2
}

But, I still don't know why this would affect my tests when run individually vs all together. Also, it seems that the robolectric gradle plugin isn't necessary anymore since I'm on Android Studio 1.2, but I couldn't figure out how to get it to work without it and setting the maxParallelForks property manually.

Solutrean