How to define an experiment for first time users in Firebase?

Question

I am trying to create an experiment using Firebase remote config. The criteria is as follows: It should target only new users who have not used the app ( opening the app for the first time)

Now on further research I found that there is a user property as below:

However, this is not available in the experiment window or a similar property that fulfils the above criteria in the Firebase console as seen below:

I can only see the user properties set by my code. One way I can think of is to use one of my custom user property which is not yet set (but set to a value like null), but I don't know how I can do this.

References

https://support.google.com/firebase/answer/6317486?hl=en https://firebase.googleblog.com/2016/10/better-user-targeting-with-firebase.html

Fluxmeter

Alternative that works -> Cloud Functions that gets triggered when first open event is fired and then send a Notification to such users. – Fluxmeter

Yeah we have this problem too, seems that there is no such function out of the box for Remote Configs experiments. For the experiments only for new users we usually setup extra User Property specifically for the targeting of this experiment. Something like 'example_experiment_enrolled' and you setup enrolled/not_enrolled on the client side (and keep it in preferences for example) and create targeting with matching these criteria. – Broadminded

Answer 1

Firebase has finally made it possible. https://firebase.google.com/docs/remote-config/config-analytics?authuser=0&hl=en#target_first_open

Fluxmeter

Answer 2

I will share our experience with experiments for new users and how the different combinations of targeting work because we worked with it a lot already. You can skip the investigation part if you're not interested and just check the solution that we use now.

Our Investigations Part: Firstly, when we integrated Firebase, we were afraid that Audiences won't work properly for experiments targeting cuz all current users will be treated as new ones after integration, so we checked several approaches and went with an approach of creating specific User Properties that we specified on the client side differently for old/new users. For instance, we created a User Property called adv_experiment_enrolled and specified on client side values 'enrolled'/'not_enrolled', so all new users after installation of this version became 'enrolled' and old ones after updating the version just became 'not_enrolled'. And we just used that User Property as targeting in an experiment. That worked well, but it wasn't a general approach that we could use easily for all experiments and we needed to create User Properties for each new experiment.

So we've tried the Audiences approach after few months of integration that was what @jackes described here with First Open Time user property: https://mcmap.net/q/778276/-how-to-define-an-experiment-for-first-time-users-in-firebase And we've got several problems, first of all, seems they had some troubles with populating of that kind of audience and just ~3-5% of new users were getting there. We also created an Audience depending on First Open event itself and used it too, it was populating better and was close to the real number of installs that we had. But we've noticed problems with this approach as well and the biggest one was that an experiment had only 20-30% of users from that Audience. We tested it and noticed by some of our metrics that seems users are not enrolled in this experiment in their first session cuz 1) Firebase takes some time to enroll a user in the Audience and 2) Remote Config has 12h cache by default, so it wasn't really a data for most of the new installations.

A solution that seems to work well for now:
We were surprised with that Firebase does have User Property for First Open Time, but doesn't allow to use this as the targeting for experiments (It would be very helpful to solve this problem tbh), so we just decided to try our good experience with User Property targeting and apply the general approach First Open Time user property, so we've created our own custom_first_open_time especially to target installations after some specific time (we just used current timestamps for platforms in seconds).
Important notes:
- You have to set up user properties before loading a remote config.
- You have to keep this first open time on the client side persistently once you generated it (usually you use NSUserDefaults/SharedPreferences for iOS/Android for that)

Sample of experiment configuration:

Broadminded

How quickly does the user properties update at the Firebase end. We need to apply A/B policy at the very launch of out app and will already delay the loading of the app to get the remote config value according to the assigned test group. I'm afraid it won't work, if we first set a user property and immediately fetch the remote config. Will the user property update affect the A/B assignment? – Acetophenetidin

@MattiJokipii our tests showed that it will. Seems they either force sync or even send user properties for the remote config fetching, but you have to setup them before calling fetch remote configs. It worked well on our side for both iOS / Android SDKs. Sometimes we even use extra User Properties for targeting instead of audiences for this reason when we want to provide that for 1st session cuz audiences take more time to be assigned. – Broadminded

Yes, based on our experience, i can confirm that this is the way to accomplish targeting new users. – Acetophenetidin

it doesn't work for ios, its just ignores custom property – Cawnpore

@Cawnpore what do you exactly mean by it just ignores custom property? You have to make sure that you setup the values for the custom properties before actual fetching of the Remote Config. – Broadminded

@Peregreen I am not sure, at least it doesn't work for test devices, it is hard to test on real ones when test already started – Cawnpore

@Cawnpore as far as I remember, test devices always use config that you set up for them, so yes, it's a bit convoluted. I usually tested it through a dev a/b test, just created one in advance with a condition for minVersion of the app higher than your production one, verified the setup, and only after that rolled it out for all users. – Broadminded

We faced the same problem with Firebase Audience based on its own first_time_open user property not being accurate. We switched to using custom User Property for Remote Config Condition. Worked. – Omen

I tried to use this method, but smth strange happens when I start the experiment. 1) I created a new experiment, set custom_first_open_time > 1581944923450 in the targeting section 2) I used 1 of 10 remote config params for the experiment. Once I started, all users stopped obtaining any remote config param and inside the app was used params set as default in code (inside the app). – Byrd

@Byrd it seems that you used timestamp in milliseconds, while it should be in seconds if you track it in the similar manner. – Broadminded

@Peregreen Yeah, it was because of milliseconds and looks like firebase can't compare so large numbers. – Byrd

I see that the first_open_time user property is now available in the targeting section. Have you tried it @Peregreen? I understand with this we wouldn't need a custom user property. – Entellus

Hey, sorry, I moved on from Firebase experiments lately, and have not been up to date with their latest changes. If they have the default property, then likely you won't need a custom one anymore :) – Broadminded

Answer 3

Didn't tried, but it should work. Create an Audience in Firebase Console with following condition: First Open Time is greater than or equal to some date. Where the date should be the next day after you starting an experiment.

Crankcase

this didn't work for me, when I create an audience like that it doesn't populate properly. Why? – Soloma

In my experience, after creating audience A, the data will be collected from the next day of today, so you cannot use your audience immediately right after creating it. – Ulpian

Also can verify that audience cannot be used for this - too slow. – Dysgraphia