Executing python script built under different virtual environment/venv?

Question

I apologize if what I'm trying to achieve is not Pythonic - I recently moved to the language.

I have a project directory structured like so:

root
--proj1
----venv
----main.py
--proj2
----venv
----main.py

Both proj1 and proj2 run under their own virtual environments. I am trying to call proj2/main.py from proj1/main.py, whilst executing proj2/main.py under its own venv. I have tried:

import subprocess

s2_out = subprocess.check_output([sys.executable, r"..\proj2\__main__.py", "arg"])

This invokes successfully, but I am getting all manner of not found exceptions, etc. I am guessing this is the reason why.

Please let me know if there is a better approach!

Feverwort

Do you use pyenv by any chance? I know how to do this if you use pyenv. – Hath

@PeacefulJames Nope - but please elaborate I'm open to adopting it if it will help :) – Feverwort

Is there an overriding reason you need to have 2 venvs? You’re going to have fairly unreliable behavior unless you set things up correctly. i.e. if you activate venv1 you should be able to import proj2.main because things were configured correctly. Anything else is building on sand. Yet doing that is non trivial as well which is why I hesitate to think your config as is is a good path to start on. – Ritenuto

Answer 1

You can do this:

import subprocess
subprocess.call(["python_interpreter location (python.exe)", "python file"])

So you could do:

import subprocess
subprocess.call(["../proj2/bin/python.exe", "proj2/main.py"])

For Mac OS and Linux, the python interpreter path for a venv would be folder/bin/python.exe, or in your case ../proj2/bin/python.exe.

For Windows, the python interpreter path for a venv would be folder/scripts/python.exe.

You may need to include the full paths.

Another way to do this could be using subprocess.call, if you need the output:

import subprocess

output = subprocess.call("%s %s" %("../proj2/bin/python.exe", "proj2/main.py"))
print(output)

Both ways will work just fine :)

Nuclease

For a venv, python.exe should be in scripts. – Nuclease

I think it would be path/to/venv/bin/python under Linux/Mac and path/to/venv/Scripts/python.exe under Windows (if I remember correctly). But essentially this is the answer I would suggest as well. – Goodden

Worked well, I didn't realize I could specify the interpreter. Thanks :) – Feverwort

Answer 2

Hey this is not a complete answer but is how I would approach it. If you use pyenv then this would be the approach:

1. Make a separate virtualenv for each project. pyenv virtualenv 3.8 proj1 and pyenv virtualenv 3.7 proj1 or whatever the python versions are.
2. use pyenv local in each directory to link the dirs to the venvs
3. cd to each directory and in each one the python env should activate. Use pip install to install the libs to each venv.
4. Now you should have access to different python executables that use separate libs, e.g. ~/.pyenv/versions/proj1/bin/python
5. So from proj1 code you should theoretically be able to do:

import os;
os.system("~/.pyenv/versions/proj2/bin/python ../proj2/main.py")

or something like that. I have not actually tried this but I am fairly certain it would work for using separate libs.

Here is pyenv: https://github.com/pyenv/pyenv

I will try it myself tomorrow when I am not sleepy.

Hath

Thanks man, I was setting up pyenv but went with the simple solution :) – Feverwort

Answer 3

This is a perfectly reasonable use-case. When trying to use multiple tools in a workflow, each might have different dependencies.. and I don't feel python supports that this well. I usually stich multiple tools via bash/files, but if you want to call from within python, you have to do something like.

env = dict(os.environ)
env['PYTHONPATH'] = ... path to the library in the env...
args = [pathwithinenv/python, fullpath to python code, arg1,arg2]
p = Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, 
stderr=subprocess.PIPE, env=env)

Subaltern