Why does PYTHONPATH with trailing colon add current directory to sys.path?

Question

Consider a Python project like this:

foo/
    __init__.py
scripts/
    run.py
demo.sh

Under normal circumstances, attempting to import from the foo package will fail if you run the script from the root of the project, because default Python behavior is to add the directory of script invoking the Python interpreter (and not necessarily the current directory) to sys.path. (docs):

python scripts/run.py

However, I recently noticed that such imports were working on my box, and I tracked it down to some surprising behavior related to PYTHONPATH. In my Bash profile I had added a directory to PYTHONPATH:

export PYTHONPATH="/some/path:$PYTHONPATH"

If PYTHONPATH is initially empty, that (somewhat sloppy, but commonly seen) form of the command will leave a trailing colon:

echo $PYTHONPATH
/some/path:

I had always assumed that this trailing punctuation had no impact, but it appears that the trailing colon was the cause of the mysteriously successful import. Leading or trailing colons (or even a defined but empty PYTHONPATH) will result in sys.path containing an empty string before the site module loads, which in turn leads to the current working directory being added to sys.path.

Here are a Python script and a Bash script to demonstrate. I got the same behavior using both Python 2.7 and Python 3.3.

Python script: run.py:

import sys, os

pp = os.environ.get('PYTHONPATH')

try:
    import foo
    print 'OK'
    assert os.getcwd() in sys.path
    assert pp == '' or pp.startswith(':') or pp.endswith(':')

except Exception:
    print 'FAIL'
    assert os.getcwd() not in sys.path
    assert pp is None

Bash script: demo.sh:

# Import fails.
unset PYTHONPATH;  python scripts/run.py

# Import succeeds -- to my surprise.
PYTHONPATH=''      python scripts/run.py
PYTHONPATH='/tmp:' python scripts/run.py
PYTHONPATH=':/tmp' python scripts/run.py

My questions:

• Am I understanding the situation correctly, or have I somehow gone astray?

• Is this documented anywhere? I did not find anything. At a minimum, I am posting this information here in case it will help others.

• Am I alone in finding this behavior unexpected?

Namedropping

Could not replicate on windows. – Meseems

You are right about what is happening and I find the behavior unexpected and undesirable. It would be worth a bug report to python IMHO except that longstanding oddities like this can break things when fixed. – Damali

Or maybe I take that back. export "PATH=$PATH:" lets commands in the local path be executed on linux. Python is working the same way. – Damali

@Damali Thanks – that's a good test. I guess the Python behavior makes sense in light of the Unix behavior ... but they both seem counter-intuitive. Maybe there's a deep logic here. – Namedropping

yeah - just repeated your discovery and then google for "python path trailing colon" and stumbled upon your answer. Seems sub-optimal to say the least. – Claptrap

Answer 1

When modifying colon-delimited environment variables such as PYTHONPATH, PATH, CPATH, MANPATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, etc... Some of these variables place special meaning to trailing colons, while others do not.

For PYTHONPATH and PATH, I would recommend prepending (or appending) new directories in a manner that does not accidentally introduce trailing (or leading) colons if the variable was previously unset:

export PYTHONPATH="/some/path${PYTHONPATH+":"}${PYTHONPATH-}"

(In the case of MANPATH and INFOPATH, you do want to introduce a trailing colon so that man and info will include their default search directories.)

Explanation:

• ${PYTHONPATH+":"} expands to a : if PYTHONPATH is set, regardless of if PYTHONPATH is empty.

• ${PYTHONPATH-} will expand to the contents of PYTHONPATH if it is set, but if PYTHONPATH is unset, then ${PYTHONPATH-} expands to nothing --- just like the usual ${PYTHONPATH}.

  • ${PYTHONPATH-} is the same as ${PYTHONPATH-""} meaning to substitute "" (nothing) when PYTHONPATH is unset.
  • The reason I recommend ${PYTHONPATH-} over ${PYTHONPATH} here is that ${PYTHONPATH-} will not create an error when PYTHONPATH is unset and your script has executed set -u to raise errors on unset variables.

For details on the ${parameter+[word]} and ${parameter-[word]} mechanisms, see "Parameter Expansion" at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02

For details on set -u, see the description of "set" at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#set

Berti

/some/path${PYTHONPATH+:${PYTHONPATH}} won't raise an error if PYTHONPATH is unbound because the ${PYTHONPATH} on the right only gets evaluated if PYTHONPATH is bound. Also, maybe /some/path${PYTHONPATH:+:${PYTHONPATH} may be desirable if you don't want to end up with a trailing colon in the case where PYTHONPATH started out set but empty. – Aster