Makefile can't use `conda activate`

Question

I need to activate a conda environment in my makefile in order to run some python scripts, however, whenever I try to run conda activate env_name, I get the following message:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. If your shell is Bash or a Bourne variant, enable conda for the current user with

$ echo ". /Users/MY_USERNAME/anaconda3/etc/profile.d/conda.sh" >> ~/.bash_profile

or, for all users, enable conda with

$ sudo ln -s /Users/MY_USERNAME/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh

The options above will permanently enable the 'conda' command, but they do NOT put conda's base (root) environment on PATH. To do so, run

$ conda activate

in your terminal, or to put the base environment on PATH permanently, run

$ echo "conda activate" >> ~/.bash_profile

Previous to conda 4.4, the recommended way to activate conda was to modify PATH in your ~/.bash_profile file. You should manually remove the line that looks like

export PATH="/Users/MY_USERNAME/anaconda3/bin:$PATH"

^^^ The above line should NO LONGER be in your ~/.bash_profile file! ^^^

I've tried changing the shell for the makefile by adding SHELL := /bin/zsh at the top, but this doesn't fix the problem. Additionally, I need this makefile to be able to run using whatever the default shell is for the computer (some of my teammates use zsh, others use bash). It seems like no matter what I do, I can't get conda activate to work in the makefile.

What can I do to get it to work, or is this impossible?

Tinner

This problem is killing me. Tried everything but nothing works. :/ – Spouse

Answer 1

After a bit of searching around, I came up with adding this pattern to my Makefile to make conda activate work. Others may be able to simplify.

# Need to specify bash in order for conda activate to work.
SHELL=/bin/bash
# Note that the extra activate is needed to ensure that the activate floats env to the front of PATH
CONDA_ACTIVATE=source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate

py3build:
    ($(CONDA_ACTIVATE) py3.6 ; python setup.py build )

Contractor

Note that this was with conda version 4.6.11 – Contractor

To which file are you adding this? to ur baschrc? If not where is the Makefile location? Is this on linux? – Superjacent

Added to a Makefile in a project containing the setup.py file. I use MacOS High Sierra. – Contractor

This stopped working for me in conda 23.5.2 – Sexagenarian

Answer 2

You should use .ONESHELL: directive at beginning of script. This run all in the same shell.

Interlard

Answer 3

Wonderful solution by @Traveler. But keeping in mind the suggestions by both @Felipe Maza & @Tomasz Bartkowiak, adding .ONESHELL: before the recipe solves all the errors.

I use a Linux system with zsh as my shell and the current latest conda version: 4.12.0

Therefore, the recipe will be:

.ONESHELL:

SHELL = /bin/zsh
CONDA_ACTIVATE = source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate

# Create conda env from env.yml and compile and install exact pip packages
conda-pip:
    conda env update --prune -f env.yml
    $(CONDA_ACTIVATE) <<env_name>>
    pip-compile requirements/req.in
    pip-sync requirements/req.txt

conda-pip does the following:

1. Create a conda environment based on an env.yml created beforehand.
2. Activate the conda environment by replacing <<env_name>> with the environment name that is set in the env.yml.
3. Use pip-compile and pip-sync to compile and install exact pip packages to the conda environment.

Note: You still need to activate the conda environment to use it in the terminal after this because conda-pip started and ended in a separate sub-shell.

You can check more about pip-compile and pip-sync here: pip-tools

Radiosonde

Answer 4

One important thing you need to remember is that Makefile will execute each line of the recipe in a separate sub-shell so e.g. exporting PATH in one line will not affect what the command in the next Makefile line can see! Please refer to the Makefile manual - Recipe Execution:

When it is time to execute recipes to update a target, they are executed by invoking a new sub-shell for each line of the recipe, unless the .ONESHELL special target is in effect (see Using One Shell) (In practice, make may take shortcuts that do not affect the results.)

Please note: this implies that setting shell variables and invoking shell commands such as cd that set a context local to each process will not affect the following lines in the recipe.

Unload

Answer 5

This is also a minimal working solution in bash. I have tested this one in Ubuntu.

Note that use of .ONESHELL: is actually mandatory. If you ignore that flag, every command in the shell would be executed in a separate shell. In that case you would need to append all you scripts with a semicolon to make a big one-line command!

.ONESHELL:

SHELL = /bin/bash
CONDA_ACTIVATE = source $$(conda info --base)/etc/profile.d/conda.sh ; conda activate ; conda activate

train:
        $(CONDA_ACTIVATE) <<put_your_env_name_here>>
        # other stuff

Broida

Answer 6

conda activate among other things set environment variables. However, GNU Make invokes each line of recipe in a newly spawned shell. You would need to invoke conda activate in each line of the recipe.

A better way is for makefile to do conda activate and then re-run itself in the newly activated environment and only then build your targets.

Greeting

But I'm not able to run conda activate in the first place. It seems like it's not aware of the shell I'm trying to use, even when I set it beforehand. – Tinner

@matt_js Try using the full path to conda, .e.g.: /usr/local/bin/conda activate. – Greeting

Unfortunately that's not where conda is. When I run which conda I get this: conda () { if [ "$#" -lt 1 ] then $_CONDA_EXE else \local cmd="$1" shift case "$cmd" in (activate) _conda_activate "$@" ;; (deactivate) _conda_deactivate "$@" ;; (install | update | uninstall | remove) $_CONDA_EXE "$cmd" "$@" && _conda_reactivate ;; (*) $_CONDA_EXE "$cmd" "$@" ;; esac fi } – Tinner

It is unfortunate for you indeed that it is a shell function. Which file is it defined in? – Greeting

My shell uses .zshrc – Tinner

Let us continue this discussion in chat. – Tinner

Did you solve this? If so your help would be appreciated. – Theine

Answer 7

Summing up all the answers, I came out to the simple set-up in the Makefile executin python file in conda virtual env:

SHELL:=/bin/bash # change shell to bash for conda

setup: # conda setup virtenv and python run
    source ~/.bash_profile && \
    conda activate my_conda_venv && \
    python my_script.py

where the content of .bash_profile file is updated automatically after conda was installed

cat ~/.bash_profile

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/homebrew/Caskroom/miniconda/base/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/opt/homebrew/Caskroom/miniconda/base/etc/profile.d/conda.sh" ]; then
        . "/opt/homebrew/Caskroom/miniconda/base/etc/profile.d/conda.sh"
    else
        export PATH="/opt/homebrew/Caskroom/miniconda/base/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Scarce