Canary vs. A/B release strategy

Question

I was going through the different types of release strategy and was confused between the Canary and A/B strategy. Both of them seems to be similar.

Everywhere I read on Canary is "Allows to test deployment by releasing the new version to a small group of them." and on A/B is "An A/B testing strategy targets a specific group of customers."

Then where the differences lie between them and what are the use cases of both?

References: https://azure.microsoft.com/en-in/overview/kubernetes-deployment-strategy/

Metage

Answer 1

A/B test's purpose is usually to see users' response (In a way, how much they like it) to a new UI, feature, etc. But you know that the new version works. So, you actually send randomly both versions of the application to all of them. It can be 50-50, 80-20, 90-10, anything. Sometimes the functionality is not even relevant. You might want to see which version attracts more clients and stuff like that.

Canary is more focused on how well works the new feature. Or if it actually works. It usually will be 90-10, 80-20, A >> B. Never 50-50, because if it goes wrong, you don't want half of your users to have a bad experience. So you are not positive if the new version is going to work as expected.

The most important difference (and this is what almost no one talks about) is that a canary testing has session affinity. So it doesn't send both versions to all users, but randomly sends some users to the new version, and keeps them on the same version.

Boliviano

Correct me if I am wrong but won't A/B testing also be having session affinity? I know these are meant to serve two different purpose, canary testing can be a sort of test that occurs on production after having been promoted from developement and staging environments after internal testing, whereas A/B testing is just seeing how people may responsd to a feature by sending traffic to a subset of the the users. You want to release the canary deployment after getting the bug fixed but an A/B test could just be a test for how people react to change in the product they are using. – Songster

So if I were say applying an A/B test using something like istio, the method to implement would be the same only, configuration wise? – Songster

@AnshumanKumar A/B testing shouldn't have session affinity. Usually you would fix a bug in pre-production. that's why pre and pro should be the same, so you can re-produce these kind of things. Not sure if I understand Istio question. Istio can ne used for A/B testing with subsets, but session affinity is only based on http headers, because traffic comming from outside will always hit the ingress gateway. – Boliviano

Istio does support cookie based and header based session affinity,but if at a really basic level, to implement a Canary release OR two versions of the application for an A/B test, we would basically need to specify the percentage of traffic being routed to each version (using a VirtualService in the case of Istio). But if we don't have session affinity in A/B testing, then a user could end up getting a different version of the application than they used the last time, right? – Songster

I kind of got the discussion sidetracked to Istio, but the gist of it is, if we don't have session affinity in A/B testing, a user can't receive the version of the application they had opted in for, right? For example, if I am XYZ social media company releasing a "brand new" Stories feature as initially as an opt in feature, and if users opt in, they should be able to get the Stories feature each time they re-open the application. I am not denying the other points as A/B testing and Canary are meant to serve different business use cases. – Songster

@AnshumanKumar Well, that's still canary, I would say, as it's a new feature that adds the stories. but you want the users to be able to say "I want to try". It would be considered A/B if you were going to dismiss one version and leave the other. In that case, you would want them to try both versions and decide. – Boliviano

Answer 2

A/B Testing

In simple terms A/B testing is a way to compare two versions of something to determine which performs better .
In an A/B test, some percentage of your users automatically receives “version A” and other receives “version B.
It is a controlled experiment process. To run the experiment user groups are split into 2 groups. Group “A,” often called the “control group,” continues to receive the existing product version, while Group “B,” often called the “treatment group”, receives a different experience, based on a specific hypothesis about the metrics to be measured
At the end the results from 2 groups which is a variety of metrics is compared to determine which version performed better.

Canary Testing

Canary Testing is a way to reduce risk and validate new software by releasing software to a small percentage of users. With canary testing, you can deliver new features to certain groups of users at a time.
Since the new feature is only distributed to a small number of users, its impact is relatively small and changes can be reversed quickly should the new code prove to be buggy.
It is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody.
While canary releases are a good way to detect problems and regressions, A/B testing is a way to test a hypothesis using variant implementations.

Blue/Green Deployment

Blue-green deployment is a software deployment strategy which utilizes two production environments (a “blue environment” and a “green environment”) in order to make the software deployment process easier and safer.
The two production environments are kept as identical as possible, and when new code is deployed, it is pushed to the environment that is currently inactive. Once the new changes have been tested in production, a router can then switch to point to the environment where the new changes are live, making for a smooth cut-over.
One of the main benefits of blue-green deployments is disaster recovery. Because there are two identical environments for production, if new changes are rolled out to one (say the blue version) and any issues are discovered, a router can just switch back to the other environment (green version) which has the old version of the code with zero downtime.
Blue-green deployment can be used for canary testing by simply having the router direct a percentage of your traffic to new version of the code to see how it performs with live traffic, before rolling out the change to 100% of your users.

Satiable

Answer 3

I believe already enough correct information has been provided in this thread. However, there is another simple differentiation WHEN you want to use either of them, so it may help you to differ between the two as their are indeed very similiar - but they serve different purposes:

• Canary testing is concerned with production performance (version-oriented)

• A/B testing is concerned about effectiveness of new features (effect-oriented)

Happy deploying!

Travax