Firebase A/B test not counting users when activation event is used on iOS

We're using the current version of the Firebase iOS framework (5.9.0) and we're seeing a strange problem when trying to run A/B test experiments that have an activation event.

Since we want to run experiments on first launch, we have a custom splash screen on app start that we display while the remote config is being fetched. After the fetch completes, we immediately activate the fetched config and then check to see if we received info about experiment participation to reconfigure the next UI appropriately. There are additional checks done before we determine that the current instance, in fact, should be part of the test, thus the activation event. Basically, the code looks like:

<code that shows splash>

…

[[FIRRemoteConfig remoteConfig] fetchWithExpirationDuration:7 completionHandler:^(FIRRemoteConfigFetchStatus status, NSError * _Nullable error) {

    [[FIRRemoteConfig remoteConfig] activateFetched];

    if (<checks that see if we received info about being selected to participate in the experiment and if local conditions are met for experiment participation>) {

        [FIRAnalytics logEventWithName:@"RegistrationEntryExperimentActivation" parameters:nil];

        <dismiss splash screen and show next UI screen based on experiment variation received in remote config>
    } else {
        <dismiss splash screen and show next UI screen>
    }
}

With the approach above (which is completely straight-forward IMO) does not work correctly. After spending time with the debugger and Firebase logging enabled I can see in the log that there is a race-condition problem occurring. Basically, the Firebase activateFetched() call does not set up a "conditional user property experiment ID" synchronously inside the activateFetched call but instead sets it up some short time afterward. Because of this, our firing of the activation event immediately after activateFetched does not trigger this conditional user property and subsequent experiment funnel/goal events are not properly marked as part of an experiment (the experiment is not even activated in the first place).

If we change the code to delay the sending of the activation event by some arbitrary delay:

<code that shows splash>

…

[[FIRRemoteConfig remoteConfig] fetchWithExpirationDuration:7 completionHandler:^(FIRRemoteConfigFetchStatus status, NSError * _Nullable error) {

    [[FIRRemoteConfig remoteConfig] activateFetched];

    if (<checks that see if we received info about being selected to participate in the experiment and if local conditions are met for experiment participation>) {

        dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(0.5 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{

            [FIRAnalytics logEventWithName:@"RegistrationEntryExperimentActivation" parameters:nil];

            <dismiss splash screen and show next UI screen based on experiment variation received in remote config>
        }
    } else {
        <dismiss splash screen and show next UI screen>
    }
}

the conditional user property for the experiment gets correctly setup beforehand and triggered by the event (causing experiment activation and subsequent events being correctly marked as part of the experiment).

Now, this code obviously is quite ugly and prone to possible race-conditions. The delay of 0.5 seconds is conservatively set to hopefully be enough on all iOS devices but ¯_(ツ)_/¯. I've read the available documentation multiple times and tried looking at all available API methods with no success in figuring out what the correct point of starting to send events should be. If the activateFetched method uses an asynchronous process of reconfiguring internal objects, one would expect a callback method that indicates to the caller the point in time when everything is done reconfiguring and ready for further use by the application. Seems the framework engineers didn't anticipate a use-case when someone needs to send the activation event immediatly after remote config profile activation…

Has anyone else experienced this problem? Are we missing something in the API? Is there a smarter way of letting activateFetched finish its thing?

Hope some Firebase engineers can chime-in with their wisdom as well :)

Thanks

Recommended topics

Hot tags