Deployment automation

Question

I have a Lift application that is packaged as a WAR archive and must be deployed under Jetty. However, I want to be able to perform a few tasks automatically:

• Specify the target server(or collection of servers). I have multiple servers, from development to testing and production servers and I would like to be able to control the destination of a deployment with great ease.
• A destination(for instance DEVELOPMENT), may mean a collection of servers, for load balancing purposes.
• Testing phase. Basically, on every deployment I would like to run the entire set of tests and prevent deployment if the application doesn't compile or if one or more tests failed.
• The WAR archive must deployed under Jetty, again on one or more Amazon EC2 machines running Linux.(Ubuntu 12.10)

I am using SBT and I have no idea how well this would play with Puppet or something similar. How would you go about this?

Pratfall

Answer 1

I've been watching this question hoping someone would come up with a clever answer. Since they haven't, I guess I'll chime in with my own feelings on the matter.

To my knowledge there is no existing tool to automate these types of deployments and I assume that's because there are so many different scenarios to deal with. At a minimum, you've got:

1. A single dev server, where it's just a matter of copying the WAR in place and then restarting the app server.
2. A single production server, where the process is similar, but you don't want the process to disrupt your users. Considerations here include retaining session data across restarts and planning the restart for a time with low usage.
3. A small cluster where you have a load balancer over many nodes. Now things really get complicated. You could be using a variety of different LB tools (HA Proxy, NGINX, Amazon's Elastic LB) and if you care about your users experience, you need to coordinate a rolling restart of the app servers. Migrating any session data between nodes is also a concern.
4. You have a large cluster consisting of smaller clusters in different geographic regions. Here you're dealing with #3 + whatever configuration needs to be done to coordinate between the regions as well.

I suppose 1 & 2 would be the easiest to find a generic tool for, and if those were the only situations I needed to deal with, I'd probably just deploy Jenkins along with the app. It can be configured pretty easily to pull from a git branch when changes occur, build the code and restart Jetty. By the time you get to 3 & 4 though, I think the number of different tools involved and the need to coordinate them has precluded any type of standard solution. I don't think this is just an issue in the Java/Scala world either since I've seen write-ups from the Github folks on the custom tools they've built to manage deployments of their Rails app.

As for Puppet, with the caveat that I've never used it, it seems like it could be a useful tool for this type of process. You'll need some type of central coordinator to deal with a cluster and I believe that Puppet can help with that.

Jamilajamill

Answer 2

You can use our product BuildMaster to do all of these things. I noticed that you mentioned Puppet, which is important to note that this tool is designed for infrastructure automation, not software deployment and delivery.

Working in the DevOps space we have discovered that it is important to keep your software and infrastructure deployments orthogonal and synchronized through team collaboration (developers are not ops people and vice-versa).

To address your bullet points:

Deploying to multiple servers

When creating an automated deployment plan, you can specify a server group and add/remove servers from the group as necessary.

Testing phase

You can setup a workflow with any number of testing environments before production, and these workflows can include automated checks (i.e. promotion requirements) for passing tests before allowing the build to be promoted to the next environment. The built-in ones are for ensuring passing Unit Tests, but you can specify them for any command line tool output or even write a custom extension.

Jetty deployment

Copying a WAR file is as simple as deploying an artifact to the server using the SSH agent.

Anglophobe

Answer 3

watched this question, too.

And actually with the same feeling: you can't consider all possible environments, and the framework just allows you choose what you wish. So, take any pleasant deployment tool and just use it.

(And of course, SBT allows you to test, prevent packaging, and there are a lot of deployment tools that have knowledge of WAR-s and Jettyies.)

Supen

Answer 4

It is a typical DevOps scenario. I think Jenkins will help address the first two requirements, by adding a couple of open source plugins to earn more control over the jobs, like "CloudBees Build Flow" and "Node and Label parameter".

Puppet or Chef will surely help you configure a bunch of systems with software configuration ready. They are almost equivalents but Chef supports less OS platforms.

I suggest you refer to the Continuous Integration of OpenStack especially on project gating and infrastructure automation: http://ci.openstack.org/

For deployment there is urbancode uDeploy which allows you to design your automated deployment process on the friendly GUI but it is commercial for now (acquired by IBM). Actually Jenkins can do the same but needs more coding and scripting.

Kellam