Best-practice for continuous integration and deployment

Question

Continuous integration concept has just been integrated in my team.

Assume we have an integration branch named Dev.

From it derived 3 branches, one for each specific current project :

• Project A
• Project B
• Project C

First, Teamcity is configured on a dedicated server and it goals is :

Compiles and launches unit and integration tests from versioned sources from each branch including Dev

Then, of course, each project branch (A,B and C) must be tested in a cloned production environment so that UAT can be carried out.

But I wonder what frequency should we deploy on? Everytime a source code changes ?

Should we deploy only Dev that contains mix of the 3 projects after merging each one to it (corresponding to the reality in next production release) or the 3 projects independently?

If Dev is deployed, potentially future changes on Dev must not be taken in account. Indeed, there might be a new project starting called Project D and that mustn't be part of the next release. So taking Dev for integration (UAT) is risked because deployer could unvoluntary integrate content of Project D and so environment will not reveal the reality of the next release.

Other solution: we're not taking Dev but independently the 3 projects, so must there be 3 cloned production environments in parallel?

If yes, UAT couldn't be reliable since behaviour of integration environment might change very often...

Concept of continuous deployment for UAT isn't clear for me...

Careerist

Answer 1

Oh boy. You're hitting real world CD problems. Really good questions.

The answer depends a bit on highly tightly coupled the development work is on the various projects.

In my ideal situation for you would be to have a number of "effort" specific test environments. In one case, you could consider a test environment for each project. When there is a completed build of Project A, you push it into Environment A which has the latest approved / production versions for B/C and you can perform basic integration tests there. If they pass, you promote the build to an integration test environment where the latest good A, is deployed along the latest B & C for the same release. When the integration test environment is passing tests, you can promote the contents of it as a release set containing known versions of A, B, & C. That release set would be deployed to any UAT, Staging, or Production environments.

The basic idea is to give each project a degree of isolation so that it can be tested well even if the other projects are (temporarily) badly broken, while getting to full integration tests as quickly as possible. We also want to make sure that whatever we find actually passes integration tests will be promoted together. Picking and choosing project versions to release that haven't been tested together is too risky for my taste.

This is actually a topic I get to talk about quite a lot. If you don't mind, I'll list out a few presentations I've given around these topics.

1) Scaling CI for Parallel Development (co-presented with Chris Lucca of Accurev)

This talks a good about broad strategies for balancing isolation and integration. Much of it assumes the sub-projects are being merged into a common code base, but the principals can be applied to independently built and deployed modules with only a little imagination.

2) Using uDeploy with Jenkins (registration required)

This is more product focused, but shows almost exactly the idea of using an integration test environment for multiple projects, creating a release set (we call it a "snapshot") and promoting that. Our integration with TeamCity is quite similar, but I think the strategy held in there may be more important

3) Slides visualizing a multi-component pipeline:

http://www.slideshare.net/Urbancode/adapting-deployment-pipelines-for-complex-applications

Rebeckarebeka

Thanks a lot for this well-explained answer. Just one precision : you wrote : "If they pass, you promote the build to an integration test environment where the latest good A, is deployed along the latest B & C for the same release." Why mix directly latest completed build of B & C with the latest good build of A ? Did you want to mean : latest A with latest B and with approved C (from production) and iteratively test until all the 3 projects be advanced versions from release ones ? – Careerist

Pretty much. The idea is you want an integration environment where you can assemble and test the things that might go to production together. Whether that's the "latest good" of each project, or the latest good of two, and the production version of another is details. – Rebeckarebeka

The first and second link are exactly the same. – Adin

The proposed solution above will cause a proliferation of environments, should the number of components increase in your organisation. Try functionally testing each component in isolation, with mocked downstream dependencies. This will enable you to better test how each application should behave with or without the new features. It's okay to "dark" release features which aren't being used as nothing should make use of them until the upstream components request it. This means versioning your requests appropriately, and having backwards compatibility. Happy to explain more detail. – Sibyl

Thank you for great answers. Can I use Eric Minick's isolated model when I have the "Core" component and almost all of my projects depend of "Core" project. Should I launch CI pipelines for all dependent environments to test Core integration? – Liris

What is the best way to check that all projects are stiil working with new "Core"? Also sometimes I have to make changes in 2 or more projects(core and others) to implement one new feature. I thing in this case I shouldn't isolate environments and test changed projects together. Otherwise CI process will be failed because stable(old) dependent version can be incompatible with changes in core. – Liris