Distributing a python package along with module dependencies using RPM

Question

I've got several python applications consisting of scripts/modules that should be packaged and deployed as RPMs.

The trickier bit is that each application should be distributed along with all python module dependencies, and these should be used in preference to any that are installed system wide.

The target hosts for some of these RPMs have limited network access, so the RPMs should contain everything needed to run the app, rather than downloading anything at deploy time.

I've looked at packaging and distributing a virtualenv, but relocating a virtualenv doesn't seem to be well supported.

I've looked at zc.buildout, but found the documentation to be lacking. I could see how to download dependencies during development, but not how to distribute them as part of a larger application. It's possible different apps require different versions of the same module, so these shouldn't be installed system wide.

Another pain point is that any python scripts in the app need to be modified to use a different sys.path during development and after deployment, I couldn't see an obvious way around this.

Are suggestions on how best to achieve this? An ideal summary of the workflow from a developer's point of view would look like:

1. download application source
2. run script to fetch specific module dependencies if not present (perhaps using pip)
3. run script to build python app, and package it and all downloaded dependencies into RPM

The final RPM should then be installable and runnable on a host with no network access, and only a python interpreter installed.

Tenancy

You might distribute a standalone python executable - packaged in RPM? Do you need your source code to be available to users ? If not look here #5458548 – Shushan

I haven't used it, but conda's aim to handle cases like yours. Check out "Rolling your own packages" section at this link. – Boddie

You should use pip bundles or pip "wheel houses" – Implausible

Answer 1

I'd see it as two separate problems.

1. You want a repeatable install/build system for your developers.

2. You want an installer builder.

Buildout (or pip, perhaps in combination with an extra script) can take care of the first problem. Basically: "how to get the project ready for development on a fresh laptop". Ideally you'd just say python bootstrap.py;bin/buildout and be ready (same with pip/virtualenv).

Now that you have a repeatable build, you can use that as the basis for an installer. Handiest is a clean virtual machine that you use just for this purpose. Virtualbox/vagrant, for instance. Make scripts that set up the virtualbox and that install the proper dependencies in there.

The installer builder script can then make a fresh checkout of your project inside the virtualbox and do the repeatable build thingy in the location where you want to have it in the installer (/opt/yourproject, for instance).

Then use FPM to make the actual package (.deb, .rpm, whatever). Pass in FPM options that tells it about the necessary dependencies, this way you can always be sure those dependencies are installed. (Note: these are OS-level dependencies like memcached or postgres; the python dependencies should be handled by pip or buildout).

If you split up your big problem in these two smaller problems, both can be attacked separately.

Josie

One problem with FPM is, that different packages (that you want to install) might have different dependencies (e.g. one package requires requests 1.6 and another requests 2.1, etc.) and you cannot express these differences, because FPM will package a single python package as RPM and install it (globally, so to speak). It would be therefore better to install a python project and its dependencies as a single RPM, like rpmvenv. – Carpic