Firebase Remote Config A/B testing shows no results after 24 hours

Question

I configured Firebase Remote Config A/B testing for Android, and we did rollout on at least 10K devices. For some reason, I see "0 users" in my A/B test after more than 24 hours.

Firebase GMS version is: 11.8.0

Should it show A/B participants in real-time or it's ok to see 0 users after 24 hours?

P.S: We are able to get AB test variants on test devices through Firebase Instance Id, it works well.

The simplest experiment which is running has only app package as a target, with no additional filters. And it shows 0 users as well.

Shoeshine

We're experiencing the same issue right now, with 2 AB tests - none of them has an activation event, and it's been 3 days since the test was started. Any ideas? – Phenolic

check if you have configured Targeting and distribution section well enough. Sometimes it's easy to use regex and forget to use "contains regex" filter e.t.c. – Shoeshine

That's the thing. I'm targeting 100% on users (on production package name). I have no activation trigger or anything like that. I'm currently 4 days in on that latest experiment, and I only have 1 user exposed so far. For previous experiments, users would start showing up almost immediately and would sit around the hundreds after 1 or 2 days. – Phenolic

@sofakingforever check this link firebase.google.com/support/releases#december_21_2018 I had same issue and it was because of a bug in firebase's latest sdk release. Downgrade the version of firebase-config to 16.1.0. Hope this helps. – Mclaurin

@ChiragMaheshwari I think you're right! We're indeed using firebase-config:16.1.2. Will downgrade ASAP. Thanks! – Phenolic

Answer 1

Finally, we found an answer!

Maybe somebody will find it helpful:

For now, it happens (no data in Firebase remote config A/B test experiment) if you have an activation event configured for A/B test experiment.

If you have 2 different experiments, both will fail to get results even if you have "activation event" configured only in 1 of them. Additionally, remote config will not work as well, you'll be able to get only default values.

We already reported to Google about, so they'll fix it at some point I hope.

Another useful info which is really hard to get:

• How long is it ok to see "0 Total Users" in experiment I've just started?

It takes many hours before you can see any data in your experiment. We were able to see results only after 21 hours after experiment start, so if you configured everything well, don't worry and wait for at least 24 hours. It will show 0 "Total Users" for many hours after the start.

• Should I use app versionName or versionCode in "Version" field of experiment setup?

You should use versionName.

Some useful info from support:

• Firebase SDK

Make sure your users have the version of your app with the latest SDK.

• Since your experiment is with Remote Config

When activateFetched() is called, all events from that point on will be tagged with the experiment. If you have a goal or activation event that happens before activateFetched(), such as automatic events like first_open, session_start, etc., the experiment setup might be wrong.

• Are you using an Activation Event?

Make sure to call fetch() and activateFetched() before the activation event occurs.

• Experiment ID of the experiments (if support asks you about)

It's the number at the end of the URL while viewing experiment results.

This debugging log could be useful to get what is going on

Also:

The good way to check if your experiment is working now is to set it to a specific version you didn't publish yet and check logs from remote config with the fresh app install(or erase all app data & restart).

It should show different variant every time you reinstall the app, since your Firebase Instance ID changes after app reinstall/app data erase. If you see variants change - then A/B test is running well.

In your "build.graddle": don't forget to set the same versionName which you set in experiment setup.

Shoeshine

Is there any link where we can check if the issue you've reported to Google is solved or being worked on? – Walton

No, they've never fixed this. We still are getting this bug from time to time and trying to contact firebase support. Things are moving really slowly. – Shoeshine

In my case, the variants works fine.. i'm testing in multiple devices and i can see all my variants and default too in some cases, but in the firebase dashboard show me 0 users exposed. I will try with fetch and activatedFetch before my activation event. – Hypothalamus

Answer 2

In my case, I was receiving results of A/B testing but suddenly, it stopped to appear. It had continued for 7 days and then results appeared. Firebase Support manager said:

what I suspected here is just a delay in showing the result in the experiments

Additionally, she said that

With that, I would suggest always using the latest SDK version and enabling Google Analytics data sharing.

In my case, I used I wasn't using the latest SDK version, but Google Analytics was enabled for "Benchmarking", "Technical Support", "Account Specialists" except for "Google products & services". I believe these settings were enabled by default (the screenshot from Google Analytics):

Disforest