How to implement a limited feature rollout (language agnostic) to your users?

Question

I would like to know some common or best practices of rolling out a new website feature to a select group of the userbase.

The users could be, for example, based solely on a percentage of your overall user base (10%). The rollout should be customizable (configurable) and support any number of features. It would also be useful to associate rollouts to specific user roles or privileges (ACL).

So, in essence, what is an architecture that would scale reasonably well?

As for the language agnostic portion, you can either provide pseudo-code, general concepts or ideas, or snippets from your preferred language to get your point across.

Links to any examples or tutorials are welcome.

Stabler

I recall a blog post some time ago regarding Twitter's strategy. If anyone can dig that up and post a comment that'd be helpful as well. – Stabler

I recently found a github library, rollout, which implements much of this functionality. It allows you to target all users, specific users, or a percentage of users. github.com/jamesgolick/rollout – Stabler

Good discussion on this here https://mcmap.net/q/173059/-what-is-a-quot-feature-flag-quot/… – Disannul

Answer 1

My recommendation would be that, for the people who are getting the new feature, the site they are on should be as close as possible to the site that everyone will be on once the feature is public.

In other words, I wouldn't have, for example, conditional logic on a page to determine whether a button should be visible or not, if that condition would only exist during this beta period. Rather, I would determine at login time whether this person is a beta participant or not (that determination might be random, might reference their role, etc). If they are a participant, they would be redirected to a separately deployed site that is deployed from a version control branch.

Once the rollout is complete, the branch is merged in and everyone sees the new feature.

This way, you don't have to worry that the codebase the public beta users are seeing is different (in some way that could break something) from the eventual release codebase.

Tanana

Thanks for the quick response Jacob. Some quick things that crop up when I think about conditional checks on login are to ensure sessions are global across machines for properly handling redirects to the same feature branched clone. – Stabler

very nice solution, but I'm wondering how one should handle cases, where along with codebase database schema is changed. I thought for a moment about separate schema and data migrations, but when it comes to releasing certain feature to all users, data merging may be a really tough task. Have you ever stumbled upon such situation? – Quorum

@mcveat: yes, that's a potential problem. In general, that's a problem with branching applications, of course. Some database approaches, such as Rails migrations, allow you to keep the schema specification with the particular codebase, but that still doesn't help you with merging the actual data. I think I would set up a new DB with a new schema and set up a data migration path that would move users from the old schema to the new one. Beta users get moved right away, and eventually everyone gets moved and the new schema becomes the only schema. – Tanana

@cballou: if the redirect only happens at login, I think it's reasonable to start a new session whenever someone logs in. – Tanana

Answer 2

At my last job we accomplished this using the load balancer and a current revision cookie.

The load balancer set a cookie identifying the revision number of the instance that the user was using. If that cookie was already present, the load balancer would just send that request to a running instance with the corresponding revision. When we deployed a new revision, the load balancer continued to send traffic with an existing revision cookie to their original revision until that revision was no longer running or the user closed their browser. New traffic would get sent to the newly deployed revision. This allowed us to test out changes for a bit while our existing users were continuing to run on the old revision. We could also manually set that cookie to test out the new rev internally on the production environment before turning new traffic onto it. When we were comfortable that the new revision had no major problems we could bring down the old revision and all traffic would start going to the latest revision.

That setup does not support backwards incompatible database changes though. There's pretty much no way to do it where you can have part of your users on one db schema and part on another, and be able to take writes to both and then somehow merge those writes together after you've decided the new rev is ok. I mean, it's possible but it really depends on what the schema changes are exactly and how they effect your app, so you can't do it in a reliable, agnostic way on deployment. For us, we just generally tried not to do backwards incompatible schema changes. If we really did need to, we'd try to postpone the destructive part (dropping a column or table) to a later revision where we could migrate the schema and have both versions running with no adverse effect on current users.

Outstrip

Answer 3

For a random selection process I like the concept of asking each customer when they have a successful login if they want to participate in beta testing, once the total required or wanted quantity of users is reached you stop asking. In the database I tend to store what server to redirect the user to and run a standard script that moves each user to the correct location upon login.

We design our app development months in advance and avoid changes to the existing schema. The reason is very obvious, of course this is not always possible so when we have a change like this we always fully document the change when it is written and plan the migration for this field as early as possible. This way we have a battle plan of what changes are being made and we can put in place the best solution possible for us. This unfortunately does change depending on the circumstances.

We always run multiple environments, we have production, development and beta generally. This means we 1 do not mess with production services that equal money, we don't have people breaking code and pulling the service offline when optimizing.

Development uses GIT for version monitoring and users never see this as we get all sorts of weird and wonderful experiments uploaded for playing with. It also uses it's own database vs the live data.

With beta, we do migrate specific user data generally but recently we have had a better experience with duplicating the entire database and planning a specific date for the start of beta, what this does is allows users to opt out of beta and others to opt in with minimal changes required to support this option. What we generally do is migrate new data between the 2 databases once per day, new opt-ins and opt-outs only take effect from the time the data has been migrated to the other platform.

We have also had success on a small scale using the existing production database for beta testing some new functions that operated out of their own table so depending on what you're doing data wise using the same live database could be a good option.

I hope this is useful for you... good luck with your testing mate.

Discussion

Answer 4

Google website optimizer seems like exactly what you are looking for.

Escalate

No not really, Google website optimizer is more like A/B Testing but @cballou is looking for a way to expose a specific user-group to a new feature e.g a new commenting system or uploader not just a change in layout – Volant