How might I handle development versions of Python packages without relying on SCM?

Question

One issue that comes up during Pinax development is dealing with development versions of external apps. I am trying to come up with a solution that doesn't involve bringing in the version control systems. Reason being I'd rather not have to install all the possible version control systems on my system (or force that upon contributors) and deal the problems that might arise during environment creation.

Take this situation (knowing how Pinax works will be beneficial to understanding):

We are beginning development on a new version of Pinax. The previous version has a pip requirements file with explicit versions set. A bug comes in for an external app that we'd like to get resolved. To get that bug fix in Pinax the current process is to simply make a minor release of the app assuming we have control of the app. Apps we don't have control we just deal with the release cycle of the app author or force them to make releases ;-) I am not too fond of constantly making minor releases for bug fixes as in some cases I'd like to be working on new features for apps as well. Of course branching the older version is what we do and then do backports as we need.

I'd love to hear some thoughts on this.

Interested

"I am not too fond of constantly making minor releases for bug fixes..." "Of course branching the older version is what we do..." Just to be clear, are you talking about the apps or Pinax itself (or both)? – Instruct

I was referring to the apps. We'd just then target the new minor release in our requirements for the dev version and backport the requirements if we want it on a minor release of the previous release of Pinax. – Interested

Answer 1

I meant to mention that the solution I had considered before asking was to put up a Pinax PyPI and make development releases on it. We could put up an instance of chishop. We are already using pip's --find-links to point at pypi.pinaxproject.com for packages we've had to release ourselves.

Interested

Not sure where the -1 came from, anyone care to explain? To me this seems like potentially the best option, as it gives a lot more control than the ==dev thing I mentioned in my answer. – Edelweiss

Answer 2

Could you handle this using the "==dev" version specifier? If the distribution's page on PyPI includes a link to a .tgz of the current dev version (such as both github and bitbucket provide automatically) and you append "#egg=project_name-dev" to the link, both easy_install and pip will use that .tgz if ==dev is requested.

This doesn't allow you to pin to anything more specific than "most recent tip/head", but in a lot of cases that might be good enough?

Edelweiss

Answer 3

Most open source distributors (the Debians, Ubuntu's, MacPorts, et al) use some sort of patch management mechanism. So something like: import the base source code for each package as released, as a tar ball, or as a SCM snapshot. Then manage any necessary modifications on top of it using a patch manager, like quilt or Mercurial's Queues. Then bundle up each external package with any applied patches in a consistent format. Or have URLs to the base packages and URLs to the individual patches and have them applied during installation. That's essentially what MacPorts does.

EDIT: To take it one step further, you could then version control the set of patches across all of the external packages and make that available as a unit. That's quite easy to do with Mercurial Queues. Then you've simplified the problem to just publishing one set of patches using one SCM system, with the patches applied locally as above or available for developers to pull and apply to their copies of the base release packages.

Maller

Answer 4

EDIT: I am not sure I am reading your question correctly so the following may not answer your question directly.

Something I've considered, but haven't tested, is using pip's freeze bundle feature. Perhaps using that and distributing the bundle with Pinax would work? My only concern would be how different OS's are handled. For example, I've never used pip on Windows, so I wouldn't know how a bundle would interact there.

The full idea I hope to try is creating a paver script that controls management of the bundles, making it easy for users to upgrade to newer versions. This would require a bit of scaffolding though.

One other option may be you keeping a mirror of the apps you don't control, in a consistent vcs, and then distributing your mirrored versions. This would take away the need for "everyone" to have many different programs installed.

Other than that, it seems the only real solution is what you guys are doing, there isn't a hassle-free way that I've been able to find.

Ori

This would be more relevant to the release process of Pinax. We have a pretty well established system there now. However, we have considered trying pip's bundles for releases. It is still on the table, but we haven't made any decisions towards moving that way. Especially since pip bundle is a very experimental feature. – Interested

Yeah after rereading your question I saw I wasn't answering it directly. – Ori