Fabric + buildout as opposed to Fabric + pip + virtualenv

Question

I've recently started playing around with Mezzanine, a django-based CMS. I recently just managed to configure Fabric to get it uploading to my host, webfaction.com, as its a bit more involved automatically creating the website on the shared hosting, and I wanted to automate that process.

Altogether, that system uses Fabric for template uploads of config files, and pip + virtualenv for handling the python packages.

However, I just recently read about buildout, and how some people swear by it for deployment, and others don't. See here: Django remote deployment with buildout and Fabric and here: http://labs.creativecommons.org/2011/07/29/not-panicking-switching-to-virtualenv-for-deployment/

While I've googled and found a ton of results for buildout vs. pip, there's not much information about buildout + fabric vs. pip + fabric. It seems like some of the features of buildout (uploading config templates, handling supervisor) can be done through fabric. Can someone tell me the advantages and disadvantages of either approach?

Note: As I'm using shared hosting for the foreseeable future, I can't sudo, which it seems buildout may require for a number of existing recipes.

Corking

There's one close-this-question ("too opinion-based") suggestion on this question right now. I think the question is fine, however. The combination with Fabric means the regular buildout/pip trade-off is different. Answers can address the trade-offs. – Blindage

Answer 1

Summary: Pip only installs python packages and there's more you need to do, obviously. You can do most of the extra work in buildout with the advantage that buildout does it for you both locally and on the server. Fabric has to do less, that way. The drawback is buildout's extra complexity, so if a couple of custom fabric commands is enough for you, that might be preferable for you. So: how does the trade-off work for you?

The long version:

Pip is good at installing python packages for your project. Buildout is good at setting up almost everything for a project (including python packages). That's the difference in goals.

Now... you bring fabric into the mix. With pip+fabric, you can call pip from within fabric to grab all the python packages and then you use fabric itself to set up everything else. An apache/nginx config file, creation of a couple of directories ("var/log/"), etc.

With buildout+fabric, you'll have configured buildout already to do a lot of the things like creating directories and generating files from templates and setting up supervisor and setting up a cronjob to fire up supervisor upon @reboot. So the fabfile has to do less.

So... you swap responsibilities. Everything you can do in buildout, you can do in fabric. Everything you can do in buildout, you can do with custom python (or shell) scripts in combination with pip ("read the README for the extra commands you have to do").

Buildout is a good place to do things if it is an integral part of your project. Think about it like this: if you need it both in production on the server and locally on your development machine, you're better off doing it in buildout. Otherwise you have to run fabric on your local machine, too. You can do it, but...

I use fabric in combination with buildout, myself. Buildout is for setting up the project itself, fabric for everything around it. Some examples:

• Actually cloning the buildout from git on the production server.

• Git pull (and checkout of the proper tag).

• Restarting supervisor.

My suggestion: look on pypi for buildout recipes to see if they are handy for you. Do they save you enough work to make it worthwhile to dive into the extra complexity that a full buildout configuration means? If you don't get enough out of buildout, you might be better off with just fabric+pip and a bunch of custom commands in your fabric file.

Blindage

Note: I've answered the question slightly differently on my blog: reinout.vanrees.org/weblog/2013/09/19/fabric-pip-buildout.html, but the idea is the same. – Blindage

Thanks for the answer! I'm curious - how would you respond to the criticisms of using buildout (one that stood out to me is that it tears down your whole environment if it fails at any point) in the second link I posted (creativecommons)? – Corking

A buildout config has multiple parts. If one part fails, whatever you installed in that part might be broken. Depends on what you're doing. Some recipes/parts first remove the old stuff, true, before installing new stuff. I guess the same can happen in different ways with pip/virtualenv. Same with packages not downloading. Truth be told, pip/virtualenv is easier to hack/circumvent with some manual steps. Those manual steps are exactly what you do not want on your production machine: great that you can save your ass with them, but do you remember the temp hack one month later? – Blindage

After reading the post I got really excited, but than I went to buildout.org and now I'm confused, and maybe disappointed. The project seems completely abandoned. The introductory video linked from the homepage is hosted on Google Video, which was closed years ago. It looks like last time anybody touched the documentation was 3 years ago (see svn.zope.org/buildout-w.... I love the idea of Buildout, but I don't feel save using it now, seeing it looks like abandonware. Am I missing something? If not, are there any good alternatives? – Isaak

It is not abandonware, but it moved from zope's svn to github: github.com/buildout/buildout :-) See github.com/buildout/buildout/graphs/contributors for the activity. The bad news is that the buildout.org page is pretty indicative of the state of buildout's documentation. The pypi page has lots of info, though. Best bet: look at existing projects that have a buildout and copy/paste them. That's the handiest way to get started. – Blindage

The other issue with buildout is it doesn't support wheels wheel.readthedocs.org. With pip, packages are cached on first install and thus deployment is very fast. – Tedford

about the wheels comment, see: docs.buildout.org/en/latest/topics/extensions.html – Paraformaldehyde

Answer 2

Take a look at fabtools which adds a lot of nice buildout functionality into your fabfile. I've worked with all sorts, Chef, Puppet (sledgehammer for a walnut) Ansible and Fabric. I find Ansible great for devops teams who are stuck, but don't want to learn a language, but personally, a well organised Fabric project wins hands down.

Foxworth