Using subprocess in anaconda environment [duplicate]

Question

I use Python 3.6.6 :: Anaconda, Inc.

I try to use subprocess to call other python script.

subprocess.run("python -V", shell=True)

I tried this code but outcome is

Python 2.7.12

My local python get caught

I also tried

subprocess.run("bash -c 'source /home/hclee/anaconda3/envs/calamari/lib/python3.6/venv/scripts/common/activate
/home/hclee/anaconda3/envs/calamari && python -V && source deactivate'", shell=True)

but I got the same result

Materialism

you default python alias is set to the system python, which is Python 2.7.12, if you want to change your default python to 3.6.6, you would have to change the python alias. – Dogberry

Anaconda has by default python 2.7.12 but in also includes python 3.6.6. If you want to choos python 3 then you may create and environment like conda create -n myenv python=3.6 and then, to use the environment you need to "activate" it. type conda activate myenv. Then if you execute python -V you should get python 3.6.6 When you are finish you may use `conda deactivate' to exit the environment. – Marmion

can you post the output of env and subprocess.run("env")? – Orelie

@Marmion that statement is not correct. It depends on which version of Anaconda you install. As you can see here there is one download with a default python3 version and a different download with a default python2 version – Skylight

@Skylight thanks for the information. You are right. It depends on the installation package selected. – Marmion

@NilsWerner My output of subprocess.run("env") PYTHONPATH=/home/hclee/calamari PYTHONIOENCODING=UTF-8 – Materialism

Answer 1

Run source activate root in linux, or activate root in Windows to activate the environment before running your code.

If this does not help you, try for a quick and dirty fix e.g.:

subprocess.run('bash -c "source activate root; python -V"', shell=True)

The reason you need to call bash is that python's run will not call the bash environment, but another which is a bit more constrained and does not contain source, so here we need to call bash... But as mentioned, if you need this command, either you are doing something special, or something is wrong with your environment...

deactivate is not needed, it does nothing cause the shell it was run on will be destroyed...

Note: For newest conda versions, the provided code will work, but there are also these options that work similarly:

conda deactivate:

subprocess.run('bash -c "conda deactivate; python -V"', shell=True)

conda activate root or base:

subprocess.run('bash -c "conda activate root; python -V"', shell=True)
subprocess.run('bash -c "conda activate base; python -V"', shell=True)

Ladonnalady

I beleive that conda basic environment (if none are created) is called base. I just checked that you can either use conda activate env or conda activate root or source path-to-conda/env-name/bin/activate – Marmion

I am running linux mint (Ubuntu), so can only speak for it, but just tried running it and it accepted 'root' but did not recognize 'base'. Also, I could not do conda activate: i.sstatic.net/CK4UN.png – Ladonnalady

what version of conda do you have? The changes mentioned by @Marmion occurred in conda 4.4. – Carburetor

It works in my terminal but It didn't work my pycharm. Other modules that I import is from my anaconda env. But subprocess is from "/pycharm-community-2018.1.4/helpers/typeshed/stdlib/3" Idk what's going on here – Materialism

@Carburetor my conda version is 4.5.4 – Materialism

@hochan: Did you set up the correct interpreter in pycharm? #19679650 – Ladonnalady

@Ladonnalady I check my interpreter in pycharm. I use conda env. I check my tenosrflow version, The version info printed that I installed. It's weird... – Materialism

@Materialism :S weird... one thing to check is what print(sys.executable) outputs – Ladonnalady

@Ladonnalady I check in my terminal and pycharm both. But print(sys.executable) outputs are the same. – Materialism

I had the same problem like @Materialism in terminal no problem to activate the environment but it fails in pycharm. So one solution could be to make the correct configuration in the Run/Debug Configurations window - any ideas what to enter here? I tried to set python=conda_python_path but although it is set it correctly by os.environ() , finally the subprocess() command didn't pick it up – Lobell

Note that pycharm has options on which conda env to run... (check properties etc) There is one property for running environment, another for the console... The configuration has also a hierarchy (you can define default, then project specifics, then run specifics...) But this is another (few) question(s) ... Search for the pycharm question(s) that relate to your needs... – Ladonnalady

Answer 2

I don't think sourcing a conda env in every subprocess call of your code is a good idea.

Instead you can find the bin dir of your current sourced env and grab the full path to binaries from there. Then pass these to subprocess when you want to call them.

import os
import sys
import subprocess

# what conda env am I in (e.g., where is my Python process from)?
ENVBIN = sys.exec_prefix

# what binaries am I looking for that are installed in this env?
BIN1 = os.path.join(ENVBIN, "bin", "ipython")
BIN2 = os.path.join(ENVBIN, "bin", "python")
BIN3 = os.path.join(ENVBIN, "bin", "aws")

# let's make sure they exist, no typos.
for bin in (BIN1, BIN2, BIN3):
    assert os.path.exists(bin), "missing binary {} in env {}".format(bin, ENVBIN)

# then use their full paths when making subprocess calls
for bin in (BIN1, BIN2, BIN3):
    cmd = ["which", bin]
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    print(proc.communicate()[0].decode())

The printed results show that subprocess is using the binaries from the conda environment instead of from the base (default) env.

.../miniconda3/envs/aws/bin/ipython
.../miniconda3/envs/aws/bin/python
.../miniconda3/envs/aws/bin/aws

Alodium

This will often work, but could break if the environment contains packages that have custom activation code. – Colorable

Answer 3

If your console/notebook is already using the correct environment, you can call subprocess with sys.executable to use current environment:

import sys
import subprocess
subprocess.run(f"{sys.executable} -V", shell=True)

Krefetz

sys.executable will give only the current folder where jupyter runed not the env_path as desribed in #36540123 – Eductive

@Eductive sys.executable returns the full path to the python.exe in the current environment, so you ensure that you're calling python with the correct version. In the example you link they split sys.executable to get the folder – Krefetz

Answer 4

The above solutions didn't work for me. What did work is:

import subprocess
output = subprocess.run('''python -c "import sys; print(sys.executable)"
                  source <your-conda-base-dir>/etc/profile.d/conda.sh
                  conda activate <name-of-the-conda-environment-you-want-to-activate>
                  python -c "import sys; print(sys.executable)"
                  conda env export''', executable='/bin/bash', shell=True, capture_output=True)
print(bytes.decode(output.stdout))

Substitute <your-conda-base-dir> by the output of which conda executed in a the interactive terminal you normally use for executing conda commands, but remove /condabin/conda from the end of the output you get, such that the /etc/profile.d/conda.sh location will be found when the above subprocess.run() is executing.

Umont