How does sys.executable determine the interpreter path?

Question

I've installed python on mac with homebrew. Some tools (pipenv for example) have some troubles because they try to write to /lib/python3.7/site-packages/, not permitted under mac. After some investigation I found that they start a new python interpreter found in sys.executable that is effectively inconsistent with the python path installed by homebrew.

$ which python
/usr/local/bin/python
$ python -c "import sys; print(sys.executable)"
/usr/local/opt/python/bin/python3.7

I would expect that these path points to the same binary, why is it not so? How can I control sys.executable?

Embayment

Is /usr/local/bin/python a symlink? Shouldnt you be using sys.argv[0] ? docs.python.org/3/library/sys.html#sys.argv – Affectation

Answer 1

It is determined at runtime (by the calculate_program_full_path() function in Module/getpath.c, to be exact. It is usually based of the argv[0] value that the OS passed in.

You can set an alternative value by setting the PYTHONEXECUTABLE environment variable.

However, on homebrew builds, a bit more is going on. Homebrew forces the issue and sets sys.executable directly in a sitecustomize.py module generated at install time:

$ tail -n2 /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/sitecustomize.py
    # Set the sys.executable to use the opt_prefix
    sys.executable = '/usr/local/opt/python/bin/python3.7'

This blithely ignores PYTHONEXECUTABLE even if set.

So what is going on, why does homebrew clobber sys.executable?

Homebrew Python is a MacOS framework build, so you can run GUI apps with this Python binary. A binary inside a framework bundle is placed under very strict requirements by Apple as to what you can do with it, including what the executable name is allowed to be set to. To work around these the framework binary is actually a wrapper ) that translates takes to a better path, sets the __PYVENV_LAUNCHER__ environment variable and launches the actual python binary located at Resources/Python.app/Contents/MacOS/Python, which then uses the __PYVENV_LAUNCHER__ environment variable to inform sys.executable.

The path the wrapper sets has any symlinks in the directory name resolved. Since homebrew makes /usr/local/opt/python a symlink to a specific Python bottle directory, running /usr/local/opt/python/bin/python3 results in sys.executable being set to the linked bottle path:

$ /usr/local/opt/python/bin/python3 -S -c 'import sys; print(sys.executable)'
/usr/local/Cellar/python/3.7.0/bin/python3

which defeats the purpose of the symlink, and can lead to broken pip-installed scripts each time homebrew makes a minor version update to the Python formula.

I'd like for homebrew to at least check if PYTHONEXECUTABLE is set here. You can force the issue yourself by just setting sys.executable directly:

import os, sys

if 'PYTHONEXECUTABLE' in os.environ and :
    sys.executable = os.environ['PYTHONEXECUTABLE']

I've opened a report to request the homebrew Python formula checks for PYTHONEXECUTABLE and included a suggested fix. The fix was landed on November 28, 2018, so just an update of your Python packages should get you the new version and make Homebrew Python honour PYTHONEXECUTABLE once again.

Interconnect

Thank you for the thorough answer, I never imagined this is so complicated. Meanwhile, I figured out that the fact that pipenv tried to write to /lib/... was caused by a setup.cfg left in the project folder from a previous version; removing that makes working everything like charm. But anyway it is a very interesting topic I learned something new. – Embayment

@MrTJ: right, had you mentioned you had a problem with pipenv I could have focused on that; that's a new problem on top. :-) – Interconnect

@MrTJ: not that I have heard of setup.cfg affecting pipenv to that extend; were you using pipenv install -e . to include the current setuptools-configured project? – Interconnect

Homebrew now respects PYTHONEXECUTABLE – Pro

@StefanDragnev: thanks for the reminder, I meant to update this with the fact that my pull request was accepted and landed. – Interconnect

now we should persuade the venv module maintainers to set the PYTHONEXECUTABLE on virtualenv activation. – Capuche