Activate conda environment in docker

Question

I need to activate environment in docker and run a command in this environment. I create the environment, but then I try to activate this environment and run the command in this way:

CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]

but when I ran docker I get an error:

[FATAL tini (8)] exec source activate mro_env && ipython kernel install 
--user --name=mro_env failed: No such file or directory

This is the whole Dockerfile:

FROM continuumio/miniconda3

ADD /src/mro_env.yml /src/mro_env.yml
RUN conda env create -f /src/mro_env.yml

# Pull the environment name out of the mro_env.yml
RUN echo "source activate $(head -1 /src/mro_env.yml | cut -d' ' -f2)" > ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 /src/mro_env.yml | cut -d' ' -f2)/bin:$PATH

CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]

Underplay

Please show us your Dockerfile; it looks like you didn't actually wind up creating the mro_env environment. Please also check out the official documentation for the CMD instruction. – Tenedos

@JordanSinger I added the Dockerfile – Underplay

Are you sure your environment creation succeeded? – Southpaw

@Southpaw Yes. the environment installed successfully. the problem is the integration with jupyter notebook. my goal is to install the environment on jupyter, and launch jupyter with this environment. Do you have any idea how could I do this? – Underplay

The way I do this is usually to install Jupyter in the environment, then launch it from there. Otherwise you could have a look to nb_conda_kernels to reach this environment from your Jupyter server. – Southpaw

I tried to installed Jupyter in the environment, then launch it from there, but I didn't succeed to launch the jupyter from the dockerfile. Do you have any idea how should the docker file looks like in order to launch Jupyter with the environment? – Underplay

Consider adapting this answer. – Clang

when using docker do we always have to create a new conda env each time we create a container? Or just once when creating the docker image? – Giffie

Using conda in docker is somewhat like gambling still. – Spanos

I've seen ppl do something like this in the docker file: RUN conda create -n pycoq python=3.9 -y then ENV PATH="/home/bot/miniconda3/envs/pycoq/bin:${PATH}". Is there something wrong with this? – Giffie

This one worked for me. – Magnesia

Maybe, this answer can help you – Risorgimento

Answer 1

Followed this tutorial and it worked. Example Dockerfile:

FROM continuumio/miniconda
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]

EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "-n", "myenv", "python3", "src/server.py"]

Update:

You can use "conda run --no-capture-output" to not buffer IO if you use the 4.9 version of conda. Updated Dockerfile:

FROM continuumio/miniconda3
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "--no-capture-output", "-n", "myenv", "/bin/bash", "-c"]

EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "myenv", "python3", "src/server.py"]

Pedi

The tutorial that you mentioned here (and the whole website) is worth every minute spent on reading it. Thank you. – Tapster

SHELL command does not work and makes the rest of my code think its in the quotes – Centigram

Last I knew, conda run buffers all standard output/error until the process is complete, so this isn't an option for an application interacting over standard I/O. I suppose it could still run a server, but don't expect to monitor any stderr messages while it's running. Also, that "tutorial" has issues, like the first example only fails because the author doesn't properly chain their commands in a single RUN (i.e., use &&), and unnecessarily running conda init when the base image already has done that setup. – Clambake

What about Docker is so ephemeral that commands need to be chained like that – Morbilli

@Clambake You can use "conda run --no-capture-output" to not buffer IO if you use the 4.9 version of conda – Pedi

when using docker do we always have to create a new conda env each time we create a container? Or just once when creating the docker image? – Giffie

I had to also added the -u flag to the python command; like so ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "myenv", "python", "-u", "src/server.py"] to get output from print statements in real time. See https://mcmap.net/q/21852/-why-doesn-39-t-python-app-print-anything-when-run-in-a-detached-docker-container – Chor

what does the SHELL command do? I read docs.docker.com/engine/reference/builder/#shell but I don't get it. Why do you need it? "The SHELL instruction allows the default shell used for the shell form of commands to be overridden. The default shell on Linux is ["/bin/sh", "-c"], and on Windows is ["cmd", "/S", "/C"]. The SHELL instruction must be written in JSON form in a Dockerfile." – Giffie

I've seen ppl do something like this in the docker file: RUN conda create -n pycoq python=3.9 -y then ENV PATH="/home/bot/miniconda3/envs/pycoq/bin:${PATH}". Is there something wrong with this? – Giffie

Answer 2

For me, the solution introduced here worked seemlessly:

FROM continuumio/miniconda3
RUN conda create -n {env} python=3.6
RUN echo "source activate {env}" > ~/.bashrc
ENV PATH /opt/conda/envs/{env}/bin:$PATH

UPD: 'env' is just an example, adjust 'env' name if needed/required.

Septuor

thank you ! i was looking for this desperately. what's the purpose to add the env in path in the last line ? – Spritsail

@Spritsail - Adds the env's bin directory to the path – Curettage

Maybe stating the obvious, but if your environment has a different name don't miss to adjust the path for the env var: /opt/conda/envs/"your_env"/bin:$PATH – Hicks

@Curettage I meant, as we activate the env before, is it not enough to acces the env package ? what does adding the env to the path add ? – Spritsail

@Spritsail I get your point. I am not sure if this is really required. I would need to test if it works out without the line. Otherwise, it doesn't hurt at least. ;) – Septuor

Answer 3

The problem for me was that running the command conda activate env inside docker after installing caused conda to ask me to use the conda init bash command. However, this command asks you to restart the shell, which we don't want to do inside docker. So the solution is to realize that the reason conda is asking you to restart the shell is because it has modified and wants to reload the contents of ~/.bashrc. We can do this manually and forego the need for restarting the shell using:

. ~/.bashrc

Here's the full dockerfile for those who want it:

FROM ubuntu:18.04
# update apt and get miniconda
RUN apt-get update \
    && apt-get install -y wget \
    && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh


# install miniconda
ENV PATH="/root/miniconda3/bin:$PATH"
RUN mkdir /root/.conda && bash Miniconda3-latest-Linux-x86_64.sh -b

# create conda environment
RUN conda init bash \
    && . ~/.bashrc \
    && conda create --name test-env python=3.7 \
    && conda activate test-env \
    && pip install ipython

Edit - deal with comment

The commenter is correct in that the above example doesn't work for switching conda environments within docker. More recently, I figured out that to get this to work, you should start every RUN command with conda init bash && .~/.bashrc && conda activate env.

Here's another example:

FROM quay.io/pypa/manylinux2014_x86_64

RUN yum install -y wget
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
RUN bash Miniconda3-py39_4.9.2-Linux-x86_64.sh -b -p /Miniconda3

RUN /Miniconda3/bin/conda create -y --name py37 python=3.7 pytest
RUN /Miniconda3/bin/conda create -y --name py38 python=3.8 pytest
RUN /Miniconda3/bin/conda create -y --name py39 python=3.9 pytest

RUN /Miniconda3/bin/conda init && bash ~/.bashrc && . ~/.bashrc

ENV conda /Miniconda3/bin/conda
ENV bashrc /root/.bashrc

# install numpy in each env
RUN $conda init && . $bashrc && conda activate py37 && pip install numpy 
RUN $conda init && . $bashrc && conda activate py38 && pip install numpy
RUN $conda init && . $bashrc && conda activate py39 && pip install numpy

Apochromatic

put the last compound command on different RUN lines and it breaks altogether. this is not a working solution if you need to switch between different environments in a container. – Morbilli

@Jeremy Leipzig is correct. In order to switch between conda environments in subsequent RUN commands, you will need to start every RUN command with conda init && . ~/.bashrc && conda activate py38 (for example). – Apochromatic

Answer 4

You can set CONDA_DEFAULT_ENV

Like this:

FROM continuumio/miniconda3

ARG conda_env=mro_env

ADD /src/environment.yml /src/environment.yml
RUN conda env create -f /src/environment.yml

ENV PATH /opt/conda/envs/$conda_env/bin:$PATH
ENV CONDA_DEFAULT_ENV $conda_env

CMD [ "python", "test.py" ]

UPDATE:

Better use activate. Work for me:

FROM continuumio/miniconda3

ADD /src/environment.yml /src/environment.yml

RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env"

CMD [ "python", "test.py" ]

Picard

It is prepending the conda env's bin that can work, not sure what the role of CONDA_DEFAULt_ENV is here. And this could mess conda activate as you do not define other variables as CONDA_SHLVL etc... – Southpaw

My problem is launching jupyter with this environment. Do you have any idea how to install the environment on jupyter and launch it with the environment? – Underplay

Specifying /bin/bash -c "source activate mro_env" seems entirely pointless since the active shell for the ENTRYPOINT and/or CMD that follow is not bash; it is evidently sh. Setting CONDA_DEFAULT_ENV seems to also similarly be pointless. In both cases, the real work is being done by ENV PATH. – Clang

when using docker do we always have to create a new conda env each time we create a container? Or just once when creating the docker image? – Giffie

I've seen ppl do something like this in the docker file: RUN conda create -n pycoq python=3.9 -y then ENV PATH="/home/bot/miniconda3/envs/pycoq/bin:${PATH}". Is there something wrong with this? – Giffie

I don't think this works => ERROR [7/8] RUN /bin/bash -c "conda activate pycoq" 0.6s ------ > [7/8] RUN /bin/bash -c "conda activate pycoq": #11 0.518 #11 0.518 CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. #11 0.518 To initialize your shell, run – Giffie

Answer 5

As the user merv points out in one of the comments above (sorry don't have enough rep to vote up the comment) conda run buffers all stdout/stderr, thus making it not workable for applications interacting or even just displaying logs over I/O.

I noticed there was no accepted answer, so I just post what has worked very well for me:

You can use an entrypoint script to activate the conda enviroment and let it take over further input from the Dockerfile such that the python script can be executed within the activated conda environment

Example Dockerfile:

FROM continuumio/miniconda3

# make docker use bash instead of sh
SHELL ["/bin/bash", "--login", "-c"]

# copy all necessary files
COPY environment.yml .
COPY ownchain/mypyscript.py .
COPY entrypoint.sh /usr/local/bin/

# make entrypoint script executable
RUN chmod u+x /usr/local/bin/entrypoint.sh
# create environment
RUN conda env create -f environment.yml

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["python", "mypyscript.py"]

Where the entrypoint.sh looks like this:

#!/bin/bash --login
set -e

# activate conda environment and let the following process take over
conda activate myenv
exec "$@"

All credit to David R. Pugh from this post that has more details, in particular with regards to Jupyter.

Seraphina

when using docker do we always have to create a new conda env each time we create a container? Or just once when creating the docker image? – Giffie

I've seen ppl do something like this in the docker file: RUN conda create -n pycoq python=3.9 -y then ENV PATH="/home/bot/miniconda3/envs/pycoq/bin:${PATH}". Is there something wrong with this? – Giffie

Answer 6

Here is how to run a script in a conda environment without buffering all standard input/output.

The --no-capture-output option is available since conda version 4.9.

FROM continuumio/miniconda3

COPY ./environment.yml ./environment.yml

RUN conda env create && conda clean --all -f --yes

ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "the_environment_name", "python", "myscript.py"]

Perfidy

Answer 7

I understand that there is no one solution fits all but this is what I have been using with my Flask applications:

FROM continuumio/miniconda3

COPY environment.yml .
RUN conda env create -f environment.yml

COPY app /app
WORKDIR /app

CMD ["conda", "run", "-n", "my_env", "python", "app.py"]

The logic is very simple. First, environment file is copied followed by the creation of the environment. Next, application files (this is where all my Flask application files are located) are copied. Finally, using CMD the application is started by pointing to the environment.

This is the project directory structure I used with the Dockerfile:

-- project
    -- app
        -- app.py
    -- environment.yaml
    -- Dockerfile

Note that this does not activate the environment per se but points to it at run time in the CMD command

Plimsoll

Answer 8

I stumbled upon this question while trying to activate an environment and then installing some packages inside it. The SHELL solution did not work for me (maybe because I was attempting this on an older version of conda - 4.5.4 to be precise).

The solution is to activate the environment and run all required commands inside a new shell. Also, remember that each RUN will start a new shell that will not remember anything from the previous RUN.

FROM continuumio/miniconda3

ADD /src/environment.yml /src/environment.yml

RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env \
    && conda config --add channels conda-forge \
    && conda install Jupiter \
    && conda env list"

CMD [ "python", "test.py" ]

Note every command is within "" of the same /bin/bash -c.

Pretentious

Answer 9

Similar to other answers but with SHELL which looks more cleaner

RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& echo "source activate base" > ~/.bashrc

SHELL ["/bin/bash", "-c"]
ARG CONDA_ENV=env_name 
# Create env
RUN conda --version \
  && conda create -n ${CONDA_ENV} python=3.7 \
  && source activate ${CONDA_ENV}

Adhesion

Answer 10

It is pretty hopeless to get this working reliably. Sometimes a solution works on a system for a while and then suddenly stops working. There is also little to now support for this from the continuum/minicoda3 creators. I guess, because they intend to have the container used differently. I tried the following and it runs through, however, when trying to run the container with docker-compose it looks for the environment at a weird location /opt/conda/envs/ instead of /root/miniconda3/envs/.

FROM ubuntu:22.04
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN apt-get update

RUN apt-get install -y wget build-essential time && rm -rf /var/lib/apt/lists/*

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh 

RUN conda --version

RUN conda init bash

RUN conda create -n myenv -c conda-forge -c bioconda python==3.10 pip

SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt

Spanos

Answer 11

I use this to activate an environment when I run a container. In order to install packages into that environment I just use "-n myenv".

ENV PATH /opt/miniconda/bin:${PATH}

RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh --no-check-certificate && /bin/bash ~/miniconda.sh -b -p /opt/miniconda
RUN rm ~/miniconda.sh
RUN conda clean -ya

RUN conda update -n base -c defaults conda
RUN conda create --name detic python=3.8 -y

RUN echo ". /opt/miniconda/etc/profile.d/conda.sh" >> ~/.bashrc
RUN echo "conda activate detic" >> ~/.bashrc

RUN conda install -n detic ipython

I don't like to change the entrypoints

Shrubbery

Answer 12

You can use this solution with docker-compose.

In accordance to this article, exists an easier way to do this, if you place SHELL ["conda", "run", "-n", "<venv>", "/bin/bash", "-c"] in your Dockerfile and use conda run --no-capture-output -n <venv> <your awesome command> in your docker-compose.yml.

Verticillate

Answer 13

If you don't need to change environments away from the base you could also do this:

COPY conda.yaml /
RUN { echo "name: base"; tail +2 /conda.yaml; } > /base.yaml
RUN conda env update --file /base.yaml --prune

The environment in conda.yaml could have any name since we replace it with base.

Myrna

Answer 14

Since, conda run is an experimental feature the correct way is to add this line to your Dockerfile

SHELL [ "/bin/bash", "--login", "-c" ]

after this you can continue with

RUN conda init bash

and then continue to activate your environment with

RUN conda activate <env_name>

Cindicindie