Mismatch between sys.executable and sys.version in Python

Question

There are two Python interpreters installed:

[user@localhost ~]$ /usr/bin/python -V && /usr/local/bin/python -V
Python 2.4.3
Python 2.7.6

Sudo changes PATH for every command it runs as follows:

[user@localhost ~]$ env | grep PATH && sudo env | grep PATH
PATH=/usr/kerberos/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/home/user/bin
PATH=/usr/bin:/bin

I run a test script:

[user@localhost ~]$ cat what_python.py
#!/usr/bin/env python

import sys
print sys.executable
print sys.version
[user@localhost ~]$ sudo python what_python.py
/usr/bin/python
2.7.6 (default, Feb 27 2014, 17:05:07) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]

and get path to Python 2.4.3 in sys.executable and version 2.7.6 reported in sys.version. Clearly sys.executable and sys.version do not match. Taking into account how sudo modifies PATH I can understand the value of sys.executable. However, why does sys.version report version 2.7.6 and not version 2.4.3, which would match usr/bin/python path reported by sys.executable?

This is a follow-up to my question Sudo changes PATH, yet executes the same binary

Locular

Can you use ls -l /usr/bin/python and see if possibly it's pointing to a different executable when logged in as superuser? it would be weird, but it's possible. – Potent

My guess is that sudo does a PATH search for the binary to run using the users environment before it cleans the environment, sets the new PATH, and executes it. That might be by design, because, e.g. if a user is accustomed to using gcc version X that is in /usr/local/something, they might be surprised to see a different version when they run sudo gcc. And, as you've seen, sudo resets the environment before actually running the command, which seems to maybe break some Python assumptions... – Ply

@CorleyBrigman Output of [user@localhost ~]$ sudo ls -l /usr/bin/python is -rwxr-xr-x 2 root root 8304 Oct 23 2012 /usr/bin/python – Locular

@Ply (…) which seems to maybe break some Python assumptions... – I don't think so as a) I don't suspect sudo hides information about location of executable from process it runs and b) I don't think Python has to make any assumptions about location of its executable as in normal circumstances (including sudo) every process should know location of its executable i.e. this information is not hidden from it. – Locular

@PiotrDobrogost By that I meant it looks like sys.executable searches the current PATH instead of resolving argv[0] (or maybe because argv[0] is simpy python in this case...), which was not necessarily the executable that was spawned by sudo, because sudo changed PATH before invoking python, but after it resolved which python to invoke. – Ply

@twalberg, Piotr, the correct way for a Linux program to determine its executable is to resolve the link under /proc/self/exe. If python isn't doing this and is searching PATH instead, this would explain the disparity. You could check by writing a script which shows both the target of /proc/self/exe and sys.executable or use my suggestion below and check externally while the program is running. – Rosebay

@Ply You were right – see my answer. – Locular

Answer 1

Both @Graeme

The fact that python may be unable to retrieve this suggests that it is doing its own PATH search (…)

and @twalberg

(…) it looks like sys.executable searches the current PATH instead of resolving argv[0] (or maybe because argv[0] is simpy python in this case...), (…)

were basically right. I was reluctant to believe that Python does something so simple (silly?) as using PATH to locate itself but this is true.

Python's sys module is implemented in Python/sysmodule.c file. As of version 2.7.6, sys.executable is set at line 1422 like this:

 SET_SYS_FROM_STRING("executable",
                     PyString_FromString(Py_GetProgramFullPath()));

Py_GetProgramFullPath() function is defined in file Modules/getpath.c starting from line 701:

char *
Py_GetProgramFullPath(void)
{
    if (!module_search_path)
        calculate_path();
    return progpath;
}

Function calcuate_path() is defined in the same file and contains the following comment:

/* If there is no slash in the argv0 path, then we have to
 * assume python is on the user's $PATH, since there's no
 * other way to find a directory to start the search from.  If
 * $PATH isn't exported, you lose.
 */

As can be seen in my case, one loses also when the first Python on exported $PATH is different than the Python being run.

More information on the process of calculating placement of interpreter's executable can be found at the top of getpath.c file:

Before any searches are done, the location of the executable is determined. If argv[0] has one or more slashes in it, it is used unchanged. Otherwise, it must have been invoked from the shell's path, so we search $PATH for the named executable and use that. If the executable was not found on $PATH (or there was no $PATH environment variable), the original argv[0] string is used.

Next, the executable location is examined to see if it is a symbolic link. If so, the link is chased (correctly interpreting a relative pathname if one is found) and the directory of the link target is used.

Let's make a couple of tests to verify the above:

If argv[0] has one or more slashes in it, it is used unchanged.

[user@localhost ~]$ sudo /usr/local/bin/python what_python.py
/usr/local/bin/python
2.7.6 (default, Feb 27 2014, 17:05:07) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]

Ok.

If the executable was not found on $PATH (or there was no $PATH environment variable), the original argv[0] string is used.

[user@localhost ~]$ sudo PATH= python what_python.py
<empty line>
2.7.6 (default, Feb 27 2014, 17:05:07) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]

Wrong. In this case statement from sys module's documentation is true – If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None. .

Let's see if adding location of python's binary back to the PATH (after sudo had removed it) fixes the problem:

[user@localhost ~]$ sudo PATH=$PATH python what_python.py
/usr/local/bin/python
2.7.6 (default, Feb 27 2014, 17:05:07) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]

It does.

Related:

• Python issue 7774 – sys.executable: wrong location if zeroth command argument is modified.
• Python issue 10835 – sys.executable default and altinstall
• python-dev mailing list thread – towards a stricter definition of sys.executable
• Stackoverflow question – how to find the location of the executable in C

Locular

Answer 2

I think /usr/local/bin/python is the executable that is running. The version string is almost certainly compiled into python, so it is very unlikely to be wrong. Looking at the documentation for sys.executable:

sys.executable

A string giving the absolute path of the executable binary for the Python interpreter, on systems where this makes sense. If Python is unable to retrieve the real path to its executable, sys.executable will be an empty string or None.

The fact that python may be unable to retrieve this suggests that it is doing its own PATH search using the PATH sudo has set (which as per my answer to the previous question is not the same as the one it used to find the executable).

The only way to be certain here is to dig through the python implementation, but generally I would say that the version string is more likely to be the one you can trust. On the other hand though, sudo uses execve to execute the command (at least according to the man page). You must specify the full path of the executable to execve (some of the exec variations do their own PATH search, this one doesn't). Therefore it should be a no brainer for python to fill in sys.executable.

I don't know if there is any way to get the actual argv[0] for the python interpreter (sys.argv[0] is always the name of the script or -c), but this would be interesting to see. If it is /usr/local/bin/python, this would be a bug in python.

I think the best thing to do is just to set secure_path in /etc/sudoers, hopefully then you will get some consistency.

Update

Actually execve takes on argument for the executable path and then an argv array, so argv[0] is not necessarily /usr/local/bin/python. You can still find it out though by making a script like:

import time
time.sleep(60)

Then run it and get ps to give you the full arguments:

sudo python sleep.py &
ps -o args= -C python

Also to be sure which python is being run, you can do:

sudo ls -l /proc/PID/exe

while the program is running.

Rosebay

Answer 3

Every time you start python interpreter, the shell goes to /usr/bin/python and executes it (try next: python -c "import os; print(os.environ['_'])").

Well then, as you can see yourself ln -l | grep python /usr/bin/python is a soft link to the python interpreter executable.

What I did was:

1. Install the last version of python (go to the website of python, download the last code and configure && make && make install )
2. Check location of this last version executable.
3. Erase /usr/bin/python soft link ##(you will need root permission sudo)
4. ln -s <location of last python version executable> /usr/bin/python ##(quite likely sudo needed)
5. Execute python from the command line.
6. import sys

sys.version ## it should be the last one.

Samaria