How to Install Private Python Package as Part of Build

Question

I have a fairly large private python package I just finished creating. I'd like to install it as part of my build process for an app in a Docker container (though this isn't so important). The package source is quite large, so ideally I'd avoid downloading/keeping the whole source.

Right now, I've been just passing around the package source along with my app, but this is unwieldy and hopefully temporary. What's a better way? git submodule/subtree? I'm pretty new to this.

Disbursement

Answer 1

If you use github with a private repo you will have to create a SSH deploy key and add the private key to your app folder for builds.

pip install git+git://github.com/myuser/foo.git@v123

Alternatively, you can mount a pip-cache folder from host into container and do pip install from that folder. You'd have to keep the python packages in the cache dir with your app.

pip install --no-index --find-links=/my/pip-cache/

you can install python packages to this pip-cache with the following command:

pre pip 9.0.1:

pip install --download pip-cache/ package1 package2

pip 9.0.1+ (thx for comment @James Hiew):

pip install download pip-cache/ package1 package2

Janiculum

private keys should not be kept with source, so you'd have to find a way to make the key available to the build process - I haven't done it this way. – Janiculum

pip install --download has been replaced with pip download in pip 9.0.1 – Neoplatonism

with buildkit you can forward ssh keys - medium.com/@tonistiigi/… – Janiculum

Answer 2

You can set up a personal OAuth token in Github following the instructions here and use command

pip install git+https://<your_oauth_token>:[email protected]/myuser/foo.git

More docs here.

Harper

This answer appears to be outdated. Can you please update? – Remission

what if the foo repo has submodules? How do I pass a deploy token for submodules as well? – Gettysburg

Answer 3

If you don't want to mess with ssh keys use http and token authentication, but you don't want to expose your token in your repo I suggest to pass the token as an environment variable.

Add to your Dockerfile:

ENV GITOKEN "$GITOKEN"
RUN bash -c "pip install -r <(envsubst < requirements.txt)"

Set your requirements as this:

+plotly==2.7.0
+git+https://[email protected]/githubuser/your_package.git@master

Krugersdorp

Answer 4

When you have a requirements.txt file and you want to do pip install -r requirements.txt in your Dockerfile, what I do is:

• Create a GITHUB_TOKEN env var. Put it in .env
• Add it as an ARG in your docker-compose.yml file
• Before installing your pip packages:

RUN git config --global url."https://${GITHUB_OAUTH_TOKEN}@github.com/".insteadOf "https://github.com/"

That allows to patch all URLs coming from requirements.txt

Hydrophyte

Answer 5

I'd put it in a separate private repo, then install it as a requirement using pip:

pip install git+git://github.com/myuser/foo.git@v123

Drank