Remote Config A/B Test does not provide results on iOS

Question

I have created and started an A/B Test on Firebase Remote Config 2 days ago on my iOS app with this code:

[FIRApp configure];
[FIRRemoteConfig.remoteConfig fetchWithCompletionHandler:^(FIRRemoteConfigFetchStatus status, NSError * _Nullable error) {
        // Do nothing
    }];
[FIRRemoteConfig.remoteConfig activateFetched];

I have confirmed that the test is live because on some devices I can see the test going on.

The problem is that, after two days, the Firebase Console keeps saying that 0 users have participated on the experiment. On the other hand, I've done a another test on Android with the same code and I can see activity after few hours.

Is there something I'm missing?

Edit - Pods versions:

Using Firebase (4.5.0)
Using FirebaseABTesting (1.0.0)
Using FirebaseAnalytics (4.0.4)
Using FirebaseCore (4.0.10)
Using FirebaseInstanceID (2.0.5)
Using FirebasePerformance (1.0.6)
Using FirebaseRemoteConfig (2.1.0)

Emblazon

Answer 1

Finally I reproduced the issue and found a workaround.

The key is detaching activateFetched from the application:didFinishLaunchingWithOptions: method.

So, this is the workaround (tested with Firebase 4.7.0):

[FIRApp configure];
[FIRRemoteConfig.remoteConfig fetchWithExpirationDuration:60*60 completionHandler:^(FIRRemoteConfigFetchStatus status, NSError * _Nullable error) {
    // Do nothing
}];

dispatch_async(dispatch_get_main_queue(), ^{
    [FIRRemoteConfig.remoteConfig activateFetched];
});

Emblazon

This approach creates a race condition where you're assuming fetchWithExpirationDuration completes before your dispatch_async runs. Instead, use fetchAndActivate. – Rattlebrain

Answer 2

The problem is that you're calling the activateFetched() outside the fetch method. You should call it when the completion handler returns with a success:

[FIRRemoteConfig.remoteConfig fetchWithExpirationDuration:expirationDuration completionHandler:^(FIRRemoteConfigFetchStatus status, NSError *error) {
    if (status == FIRRemoteConfigFetchStatusSuccess) {
        [FIRRemoteConfig.remoteConfig activateFetched];
    } else {
        // Manage error case
    }
}];

Disembroil

Answer 3

Hmmm... I'll be honest, it looks like you're doing everything right here.

Note that the way you've got your remote config fetching set up, it will take two sessions for the changes to take effect (you're essentially following the "Load values for next time" approach described in this blog post) so it might be that two days is the bare minimum needed to get any data, depending on how frequently people use your app. Maybe wait another day or two and see what's going on.

Also, obviously, this goes without saying, but if you just released the new version of your app with the updated libraries and everything, you might also need to wait a little while for all of your users to update to the latest version of your app,

Gate