Activate python virtualenv in Dockerfile

Question

I have a Dockerfile where I tried to activate python virtualenv so that, it should install all dependencies within this env. However, everything still gets installed globally. I used different approaches and none of them worked. I am also not getting any errors. Where is the problem?

1. ENV PATH $PATH:env/bin

2. ENV PATH $PATH:env/bin/activate

3. RUN . env/bin/activate

I also followed an example of a Dockerfile config for the python-runtime image on Google Cloud, which is basically the same stuff as above.

Setting these environment variables are the same as running source /env/bin/activate.

ENV VIRTUAL_ENV /env

ENV PATH /env/bin:$PATH

Additionally, what does ENV VIRTUAL_ENV /env mean and how it is used?

Softspoken

source ../bin/activate tried ? – Vmail

Are you running multiple python apps in the same Docker container? – Consistory

It's likely not best practice to use virtualenv in a Dockerfile since you'd ideally just install globally using the one app per container practice. However, I'm glad I happened upon this because I have a unit testing use case that requires virtualenv in a Dockerfile. It might seem odd but part of the test is for virtualenv integration. Thank you for asking this question. – Guam

re: "everything still gets installed globally". Most of the time when I see that happen, it's because someone is using the global pip. Build a venv in your Docker image, and then use thepip corresponding to the target virtualenv for installing packages into that virtualenv. If you call /path/to/venv/bin/pip (note the the full venv path) you'll likely find success. – Revisionism

Answer 1

You don't need to use virtualenv inside a Docker Container.

virtualenv is used for dependency isolation. You want to prevent any dependencies or packages installed from leaking between applications. Docker achieves the same thing, it isolates your dependencies within your container and prevent leaks between containers and between applications.

Therefore, there is no point in using virtualenv inside a Docker Container unless you are running multiple apps in the same container, if that's the case I'd say that you're doing something wrong and the solution would be to architect your app in a better way and split them up in multiple containers.

---

EDIT 2022: Given this answer get a lot of views, I thought it might make sense to add that now 4 years later, I realized that there actually is valid usages of virtual environments in Docker images, especially when doing multi staged builds:

FROM python:3.9-slim as compiler
ENV PYTHONUNBUFFERED 1

WORKDIR /app/

RUN python -m venv /opt/venv
# Enable venv
ENV PATH="/opt/venv/bin:$PATH"

COPY ./requirements.txt /app/requirements.txt
RUN pip install -Ur requirements.txt

FROM python:3.9-slim as runner
WORKDIR /app/
COPY --from=compiler /opt/venv /opt/venv

# Enable venv
ENV PATH="/opt/venv/bin:$PATH"
COPY . /app/
CMD ["python", "app.py", ]

In the Dockerfile example above, we are creating a virtualenv at /opt/venv and activating it using an ENV statement, we then install all dependencies into this /opt/venv and can simply copy this folder into our runner stage of our build. This can help with minimizing docker image size.

Consistory

The point is to save space. You can copy the virtualenv directory as is without the need of python3-virtualenv in the target image. That saves you the whole toolchain (gcc and friends) and thus a few hundred megabytes. – Sherer

You don’t need python3-virtualenv to do dependency isolation between containers. – Consistory

Many Python packages only support installation in a virtual environment, in which case it's useful to be able to activate the venv inside a docker container. – Microstructure

@MarcusLind I think the question is about packaging the contents of a python project into docker without having a build environment inside docker. Virtualenv here is used for packaging together all the dependencies in a subdirectory so you can COPY them into WORKDIR. However, this will fail because it won't handle binary dependencies which have to be built against whatever base OS his docker image is using. Instead, a solution would be to create a docker image for building the dependencies and then copying them over to target image in a multi-staged build. – Dwaynedweck

Downvoting for offtopic. If author is concerned about specific problem of usage of virtualenv together with Docker it means that he actually needs to use virtualenv with Docker. – Foraminifer

Mature counter arguments should be read on hynek.me/articles/virtualenv-lives – Icken

@GillBates But what if it answered my question coming from Google? – Stith

Venv inside docker leads to sysadmin confusion. Why use venv inside a virtual environment. Docker works for multiple languages, not just python. – Pekingese

@GillBates that's an assumption (and perpetual debate on SO). Clearly if someone was asking how to put sugar in their gas tank we wouldn't all say, look they REALLY want to know how to get sugar in their gas tank. The author's ignorance regarding docker/venv is unknown, so it's hard to tell what they REALLY want. That said, don't agree venv in containers never make sense. My case: the base image packages conflict with python install – Mcnalley

@jpmorris, the debate was created by the guy writing an answer that doesn't answer the question. Your gas tank example is a bad analogy. Why? Because a venv in a Dockerfile doesn't break anything. If anything, it prevents future breakage of the app in the even there is a change in the base image. An app shouldn't have to change because the base image was changed. A packaged app today should just work 10 years from now with absolutely no change in its codebase. That and it's a matter of size. Smaller size = less attack surface. That's why SNYK says to move a venv folder to your final stage. – Dalesman

@Dwaynedweck Instead, a solution would be to create a docker image for building the dependencies and then copying them over to target image in a multi-staged build. And that's exacty how, even in 2021, SNYK says to do it. And because it's done that way, it's still a virtual environment in a Docker image. Why? Because that venv folder for that app is totally isolated from the OS of the base image. – Dalesman

@BrandonStivers analogy isn't the point, let's change it: OP asks 'I want to know how to get gas in my car with a teaspoon' do we all say that's what they REALLY want so ask it? You seem to suggest that there's NEVER a reason NOT to use venv. How about: 'I need this container for 1 week'? "I'm fine just tagging base image (so it never changes)" etc. Point is: there are valid reasons for NOT using venv and we don't REALLY know what the OP wants. I've always been in favor of SO being answers for everyone not just the asker, so don't mark possible answers 'offtopic' and inform everyone. – Mcnalley

@jpmorris, there's NEVER a reason to NOT use a venv in final production. 12-facor-app written by a co-founder of Heroku is the industry standard. SNYK says to use them as well. But if you want final images in production (or even in staging) that are twice as large as they need to be, go ahead. Waste your bandwidth and open up security holes that don't need to be there. And good (best) practice is to use the same process in dev as you do in production. Do the math. Excuses & shortcuts aren't a valid reason to do anything. I'm just going down the street a mile... don't have to wear my seatbelt.. – Dalesman

But what if your container is based on an image with already installed packages which have conflict with the new package that you want to install. In my case, I use nvcr.io/nvidia/pytorch:21.06-py3 docker image as my base image. But then later when I want to install a python package (detectron2) I get dependency resolution conflict. – Sigridsigsmond

virtual env are preferred when using multistages build – Franklyn

So, today, in 2023 when PEP 668 is out and basically blocking pip install outside of venv by default, what is the recommendation for Docker images? I tend to agree with this answer, as I'm already in Docker, I don't really care about venv. However now it seems I'm somehow forced to... Would be great to see one more update on the answer for today's situation... – Encyclopedia

This answer is incorrect in so many ways. Basically PEP 668 explains why you should not install python packages without venv inside any OS, docker or not. downvote for making incorrect and dangerous statement bold – Jauregui

Answer 2

There are perfectly valid reasons for using a virtualenv within a container.

You don't necessarily need to activate the virtualenv to install software or use it. Try invoking the executables directly from the virtualenv's bin directory instead:

FROM python:2.7

RUN virtualenv /ve
RUN /ve/bin/pip install somepackage

CMD ["/ve/bin/python", "yourcode.py"]

You may also just set the PATH environment variable so that all further Python commands will use the binaries within the virtualenv as described in https://pythonspeed.com/articles/activate-virtualenv-dockerfile/

FROM python:2.7

RUN virtualenv /ve
ENV PATH="/ve/bin:$PATH"
RUN pip install somepackage

CMD ["python", "yourcode.py"]

Koetke

this will not work if yourcode.py creates a subprocess, I think. You also need to fiddle with $PATH, as explained in monitorius' answer. – Dantedanton

If your code creates a subprocess that invokes python use sys.executable to get the path the virtualenv interpreter. e.g. subprocess.run([sys.executable, '-m', 'foo',]) which is generally a good idea anyhow for a lot of other scenarios. – Rapscallion

Answer 3

Setting this variables

ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

is not exactly the same as just running

RUN . env/bin/activate

because activation inside single RUN will not affect any lines below that RUN in Dockerfile. But setting environment variables through ENV will activate your virtual environment for all RUN commands.

Look at this example:

RUN virtualenv env                       # setup env
RUN which python                         # -> /usr/bin/python
RUN . /env/bin/activate && which python  # -> /env/bin/python
RUN which python                         # -> /usr/bin/python

So if you really need to activate virtualenv for the whole Dockerfile you need to do something like this:

RUN virtualenv env
ENV VIRTUAL_ENV /env                     # activating environment
ENV PATH /env/bin:$PATH                  # activating environment
RUN which python                         # -> /env/bin/python

Sternpost

Another pretty popular option is to run a bash script as an entry point and let it do the rest heavy-lifting. – Avaria

Entry point is executing in runtime, when an image is already built and deployed. It should be a really special case if you want to install your packages to virtualenv while in runtime, instead of image build time – Sternpost

Answer 4

Although I agree with Marcus that this is not the way of doing with Docker, you can do what you want.

Using the RUN command of Docker directly will not give you the answer as it will not execute your instructions from within the virtual environment. Instead squeeze the instructions executed in a single line using /bin/bash. The following Dockerfile worked for me:

FROM python:2.7

RUN virtualenv virtual
RUN /bin/bash -c "source /virtual/bin/activate && pip install pyserial && deactivate"
...

This should install the pyserial module only on the virtual environment.

Precision

Thanks for the provided solution, although it did not work for me. Now, the dependency (django) is installed but I cannot find where as python 2/3 cannot import it while being outside or inside of virtualenv. I do not have a complex app, therefore I'd stick to the main purpose of Docker for now, although, there are still threads where it is explained why creating venv inside the docker container is still a fine practice. Example – Softspoken

Hope you solved the problem anyway. However that's odd, how do you check where the installation is done? – Precision

Is the "&& deactivate" at the end really needed? docker is starting subsequent RUNs in new shells anyway, right? – Dishpan

Right, I just added it to be clean in case the activation had any impact on the filesystem, which would remain in the resulting Docker image. It is most likely dispensable. – Precision

@Precision Would you maybe have any update on your answer given PEP-668 is out in 2023 and now the system basically blocks non-venv pip install? – Encyclopedia

Answer 5

The only solution that worked to me is this

CMD ["/bin/bash", "-c", "source <your-env>/bin/activate && cd src && python main.py"]

Franklyn

Answer 6

If you your using python 3.x :

RUN pip install virtualenv
RUN virtualenv -p python3.5 virtual
RUN /bin/bash -c "source /virtual/bin/activate"

If you are using python 2.x :

RUN pip install virtualenv
RUN virtualenv virtual
RUN /bin/bash -c "source /virtual/bin/activate"

Catrinacatriona

Answer 7

Consider a migration to pipenv - a tool which will automate virtualenv and pip interactions for you. It's recommended by PyPA.

Reproduce environment via pipenv in a docker image is very simple:

FROM python:3.7

RUN pip install pipenv

COPY src/Pipfile* ./

RUN pipenv install --deploy

...

Icken

Sorry if this is a silly question but how can I use the dependencies that were installed by pipenv when using the actual image? My understanding is that pipenv installs to a virtualenv with a random name. So if I pull this image, clone my repo, and try to run pipenv run pytest then it doesn't have those installed requirements accessible from my folder. Thanks – Scoundrelly

@Scoundrelly This is the good question! I personally add --system argument to the RUN from my answer. Then you can just call pytest. But this have some caveats which is about content of a system python site-packages for a particular OS: the content can be differ. So this way is not so enterprise-ready. But usable for development. For enterprise grade solution you need to set or catch the virtualenv name, imho. – Icken