What's the standard way to package a python project with dependencies?

Question

I have a python project that has a few dependencies (defined under install_requires in setup.py). My ops people requires a package to be self contained and only depend on a python installation. The litmus test would be that they're able to get a zip-file and then unzip and run it without an internet connection.

Is there an easy way to package an install including dependencies? It is acceptable if I have to build on the OS/architecture that it will eventually be run on.

For what it's worth, I've tried both setup.py build and setup.py sdist, but they don't seem to fit the bill since they do not include dependencies. I've also considered virtualenv (which could be installed if absolutely necessary), but that has hard coded paths which makes it less than ideal.

Cellulose

don't think there's a standard way to do that. – Alsatia

Why don't use requirements.txt with dependencies? Or running pip install -r requirements.txt is not acceptable? – Conjunction

I think for that you'd have to create a bundled executable, for which there are various tools. BTW, if you have "devops people", and can't just deploy yourself, that's not devops. – Velazquez

pack your project with dependencies in docker – Eudoxia

@valentjedi I certainly could, but I'd still have to bundle them. I'll investigate doing that with --target. Thanks. – Cellulose

@Velazquez Good point about devops! Changed to ops :) – Cellulose

Answer 1

There are a few nuances to how pip works. Unfortunately, using --prefix vendor to store all the dependencies of the project doesn't work if any of those dependencies, or dependencies of dependencies are installed into a place where pip can find them. It will skip those dependencies and just install the rest to your vendor folder.

In the past I've used virtualenv's --no-site-packages option to solve this issue. At one company we would ship the whole virtualenv, which includes the python binary. In the interest of only shipping the dependencies, you can combine using a virtualenv with the --prefix switch on pip to give yourself a clean environment that installs to the right place.

I'll provide an example script that creates a temporary virtualenv, activates it, then installs the dependencies to a local vendor folder. This is handy if you are running in CI.

#!/bin/bash

tempdir=$(mktemp -d -t project.XXX) # create a temporary directory
trap "rm -rf $tempdir" EXIT         # ensure it is cleaned up
# create the virtualenv and exclude packages outside of it
virtualenv --python=$(which python2.7) --no-site-packages $tempdir/venv
# activate the virtualenv
source $tempdir/venv/bin/activate    
# install the dependencies as above
pip install -r requirements.txt --prefix=vendor

Exocrine

Answer 2

In most cases you should be able to "vendor" all the dependencies. It's basically a crude version of virtualenv.

For example look at how the requests package includes chardet and urllib3 in its own source tree. Here's an example script that should do the initial downloading and copying for you: https://gist.github.com/proppy/1136723

Once you have the dependencies installed, you can reference them with from .some.namespace import dependency_name to make sure that you're using your local versions.

Frisket

Note: requests removed vendored packages some time in 2017. – Plagiary

Good answer about vendoring dependencies here: https://mcmap.net/q/263593/-import-vendored-dependencies-in-python-package-without-modifying-sys-path-or-3rd-party-packages – Plagiary

Answer 3

It's possible to do this with recent versions of pip (I'm using 8.1.2). On the build machine:

pip install -r requirements.txt --prefix vendor

Then run it:

PYTHONPATH=vendor/lib/python2.7/site-packages python yourapp.py

(This is basically an expansion of @valentjedi comment. Thanks!)

Cellulose

this --prefix vendor appears to have nothing to do with adding vendor names, this simply specifies the folder to install packages into. – Perfectionism

Correct, it's just the name of the folder to put the vendored dependencies. – Cellulose

Answer 4

let's say you have python module app.py with dependencies in requirements.txt file.

first, install all your dependencies in appdeps folder.

python -m pip install -r requirements.txt --target=./appdeps 

then in your app.py module add this dependency folder to the pythonpath

# app.py
import sys
sys.path.append('appdeps')

# rest of your module normally
#...

this will work the same way as if you were running this script from venv with all the dependencies installed inside ;>

Staid