What shebang to use for Python scripts run under a pyenv virtualenv

Question

When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?

As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.

pyenv activate venv_name
which python

Output:

/Users/username/.pyenv/shims/python

So I made my example script.py:

#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'

But when I tried running the script (from within 'venv_name'), I got an error:

./script.py

Output:

./script.py: line 2: import: command not found
./script.py: line 3: print: command not found

Although running that path directly on the command line (from within 'venv_name') works fine:

/Users/username/.pyenv/shims/python script.py

Output:

success

And:

python script.py # Also works

Output:

success

What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.

Unwrap

I use this a lot and it works fine. One example of my shebang that works fine: #!/home/dslima90/.virtualenvs/production_enviroment/bin/python, maybe check your path? Aren't you missing bin? – Hame

But if you wanto to point to your current enviroment, what is wrong with #! python ? – Hame

@Hame The last example shows that the path works fine in running the script directly, so I don't think that's it. I just tried #! python and that gives the error -bash: ./script.py: python: bad interpreter: No such file or directory – Unwrap

sorry i have an alias for my python interpreter inside my devel folder, that's why mine works (on development)... anyway #!/usr/bin/env python should do it for you... You can try on the command line before. See if it is calling the right interpreter. – Hame

@Hame Oh yeah that worked. I didn't know about that alternative to using the actual path of the executable. Thanks – Unwrap

@Hame But curious, do you know why using the path wouldn't work? – Unwrap

I really don't. Maybe something to do with your OS? Anyway I will make my comment an answer so it can also help others! – Hame

You could check with od -c script.py if the #! really are the first two characters. – Bullfrog

@Bullfrog I ran the command, and #! are indeed the first 2 chars: 0000000 # ! / U s – Unwrap

@Unwrap One possible cause is thus ruled out. Also this might be worth checking: #9988625 Is the .../shims/python another script? – Bullfrog

@Bullfrog whoah yes, you're right. It turns out it is just a bash script that calls ... exec $pyenv_python at the end. That file is created by pyenv and I just assumed it make a copy or link of a python executable, but that it is not the case. mystery solved! – Unwrap

Answer 1

I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:

#!/usr/bin/env python

That way you search your environment for the Python interpreter to use.

Hame

Confirmed that it still works in Apr. 2018 with Python 3.6.4, Thanks! – Bush

Thanks to python's "virtual shebangs" feature, this even works on Windows. – Dyadic

Unfortunately this method now conflicts with Debian Python Package policy, which has deprecated this method in favour of an explicit /usr/bin/python3 shebang. Can pyenv handle this properly? – Smallscale

@Smallscale seems like that would be great as an SO question on its own, for folks looking for a good solution for code intended as debian pkgs. But that particular issue doesn't matter for the use case from my original question, since it's just a user script and still works with current versions of python. (Also the original question was back from the bygone era when python 2 was still a thing, but not sure if that matters in the meta of things) – Unwrap

I'm coding on Windows for cross platform. On Linux, I got an error with env command: /usr/bin/env: ‘python\r’: No such file or directory -- so I had to change to newline representation in my python file on Windows and it worked. – Tull

Answer 2

If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.

#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$@"
# the rest of your Python script can be written below

Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.

You can see more here.

Darnell

@Darnell - this is awesome - I've elaborated a bit more on your suggestion below; hope that is ok – Sunda

That's a cool hack, nice work! It does have some downsides: you've set the docstring of the Python module, which you may not want, and your editor may treat the file as a Bash script, using invalid syntax highlighting. – Algiers

This will work, but be aware that PID of started and actual process will be different. This may cause some weird issues, for example if you will use such script as systemd service, notifications will not work there. – Watersoak

Answer 3

As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.

In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.

See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts

Try pyenv virtualenvs to find a list of virtual environment directories.

And then you might find a using shebang something like this:

#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'

... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:

(venv_name) $ ./script.py
success

(venv_name) $ pyenv activate non_pandas_venv

(non_pandas_venv) $ ./script.py
success

(non_pandas_venv) $ . deactivate

$ ./script.py
success

The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)

Backsaw

Knowing how to find the actual path of the virtualenv's python is handy, although this solution is missing the piece that the other 2 answers handle: i.e. being able to execute the script from any currently active virtualenv. It looks like with this method, you'd need to know ahead of time which virtualenv it's going to be executed with and could only execute it via that one virtualenv. – Unwrap

I found this question in trying to run a script dependent on the packages within a particular virtual environment while being in a different environment. In particular, I did the development at the command line, but I need to run the script from a cron job. By calling out the virtual environment's python directly in the shebang, I don't need to first setup a matching virtual environment shell and then call that environment's default python. It doesn't need to activate any virtualenv at all in the parent process. – Backsaw

Ah, I now see in your question that you wanted the generic shebang, and not a generically successful execution of a particular virtualenv-dependent script. – Backsaw

Answer 4

To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:

#!/usr/bin/env python

However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:

#!/path/to/the/venv/bin/python

Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.

Rupert

If you're using a git - and people clone it in all kinds of places, then the /path changes... it might work for you - but not for others – Articular

Answer 5

If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:

#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $@ <<EOF
import sys
print(sys.argv)
exit
EOF

I did some additional testing. The following works too:

#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python

Nudnik

The one-line works well for running scripts; cleaner than using the full path to the python interpreter, which will change with user. – Mason

Answer 6

It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:

#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3

if [ -x $PREFERRED_PYTHON ]; then
    echo Using preferred python $ALTERNATIVE_PYTHON
    exec $PREFERRED_PYTHON "$0" "$@"
elif [ -x $ALTERNATIVE_PYTHON ]; then
    echo Using alternative python $ALTERNATIVE_PYTHON
    exec $ALTERNATIVE_PYTHON "$0" "$@"
else
    echo Using fallback python $FALLBACK_PYTHON
    exec python3 "$0" "$@"
fi
exit 127
'''

__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())

Sunda

Appreciate the comments altho it's still hard to follow. Additional comments explaining the actual control flow could help. (1) starts running bash, obviously, which (2) executes python via exec, and (3) importantly the exec actually executes the original script itself via $0 – Nembutal

Answer 7

Maybe you need to check the file privileges:

sudo chmod +x script.py

Mcgrath

The comments below the question explain why it wasn't working. But my examples also showed that the execute bit had to have been set, as the output of ./script.py was not permission denied, which would have been the case had the execute bit not been set. – Unwrap