Activation event & frequency of flag updation for accurate AB test

Question

I am trying to run an experiment to change the behavior of a element on app's step-3. In order for data to be accurate,I was planning to put activation event as event which completes step-2. Now firebase remote config values are fetched when app first starts so there's no way the app would get updated remote config value for the flag to be used on step-3 if I set step-2 completion as an activation event. If I fetch and activate the flag value every-time I use a flag,that would go over limit of 5 times in 60 minutes. What is best way to go around this problem?

Onesided

Answer 1

So the activation event for ABT only causes users who trigger it to start getting counted in the experiment, it does not activate the actual config values when triggered. You should be able to fetch and activate on app start and then by using the activation event on step 2 the user may be entered into the experiment and get the step 3 behavior change.

This article talks about about RC activation https://firebase.googleblog.com/2017/01/firebase-remote-config-loading.html

And the docs here go into a bit more detail on ABT activation events https://firebase.google.com/docs/ab-testing/abtest-config

Spirillum

Thanks! Exactly what I was looking for – Onesided

Answer 2

I've been trying to do almost the same process than you and I realize that the activation event could be called without a fetch at Step 3. I'm doing the following:

Step1 (When my app starts)

Fetch variables and activate if there are any updates(.fetchAndActivate())

Step2

Just call your activation event

Step3

Get the variable that decides your variant or control group

---

What I've been using to test that this really works (This is the tricky part)

1. Remove your advertising Id
2. Remove your old app (or you can also call FirebaseInstanceId.getInstance().deleteInstanceId()) to obtain a new Firebase Token
3. Once you have a new Ads Id and firebase token, you are like a new user to FB, so you will obtain a new variant or group control.

---

Important points :

• The variants that FB gives you are fixed for your Tokens(that makes sense for Prod environments), so when you get one of the variants it will be always the same unless you reset the tokens(you only have to do this when you are testing)
• The catching time that you provide to FB is to control the fetch system locally
• The activation method only indicates to FB if this user enters or not into the experiment, but you already have a variant/control group assigned to this user.

Unruly

Also you should be aware of configuring Firebase before making your first fetch. 'FirebaseApp.configure()' This can sound stupid but if you don't do it the test A/B will fail as you will not receive the correct parameter of remote config – Kennithkennon

Thanks! The part I was confused about was the last part of the answer "The activation method only indicates to FB if this user enters or not into the experiment, but you already have a variant/control group assigned to this user.". This part wasn't clear from the documentation – Onesided

@Onesided What I mean is that whether or not you call to the activation method, FB has already assigned a Variant to this user. This means that the activation method not make a call to firebase to ask for the assigned Variant to this user, it only informs to FB that they have to take into account this user for the analytics of such experiment. – Unruly