How does virtualenv work?

Question

I checked the activate script and it looks to me all it does is:

• set VIRTUAL_ENV env
• append $VIRTUAL_ENV/bin in front of PATH

How does virtualenv provide that magical virtual environment by these? What do I miss?

Cao

Don't forget about virtualenv burrito – Indubitable

Answer 1

I will describe the basic process, which I learned from the presentation @jcollado linked to.

When Python starts, it looks at the path of the binary, and the prefixes thereof.

So let's say your virtualenv is /home/blah/scratch. The Python process knows it was executed from /home/blah/scratch/bin/python (which is usually just a copy of your system python binary /usr/bin/python) and it knows its own version X.Y because it's compiled into it. Then Python looks for lib/pythonX.Y/os.py in this order:

/home/blah/scratch/bin/lib/pythonX.Y/os.py
/home/blah/scratch/lib/pythonX.Y/os.py    <-- this file should exist
/home/blah/lib/pythonX.Y/os.py
/home/lib/pythonX.Y/os.py
/lib/pythonX.Y/os.py

It stops at /home/blah/scratch/lib/pythonX.Y/os.py because it's the first file that actually exists. If it didn't, Python would keep looking. It then sets sys.prefix based on this. It uses a similar process to set sys.exec_prefix, and then sys.path is constructed based on these.

Thetisa

X.X is derived from the Python version, so it'll be 2.7 or something similar. – Thetisa

Never thought about this path discovering for sys.prefix! – Cao

Thank you for clearly explaining in the body of your answer rather than linking to an external source which could change or disappear in the future. – Domingadomingo

Before this answer, where is the doc stating all this? Even this was very hard to find. – Irv

Answer 2

1. First the user creates a new virtualenv with the command virtualenv myenv. This creates a directory called myenv and copies the system python binary to myenv/bin. It also adds other necessary files and directories to myenv, including a setup script in bin/activate and a lib subdirectory for modules and packages.
2. Then the user sources the activate script with . myenv/bin/activate, which sets the shell’s PATH environment variable to start with myenv/bin.
3. Now when the user runs python from this shell, it will execute the copy of the binary stored in myenv/bin. Even though the binary is identical to the one in /usr/bin/python, the standard python binary is designed to search for packages and modules in directories that are relative to the binary’s path (this functionality is not related to virtualenv). It looks in ../lib/pythonX.Y where X and Y are the major and minor version numbers of the python binary. So now it is looking in myenv/lib/pythonX.Y.
4. The myenv/bin directory also contains a script named pip so that when the user installs new packages using pip from the virtualenv, they will be installed in myenv/lib/pythonX.Y

Electrophoresis

I'm wondering where do you guys know the secret path looking of something like ../lib/pythonX.Y and even other stuff related to sys.prefix and etc? – Cao

I haven't seen any good documentation about it yet, but I noticed this behavior when building Python from source. You can also test it like this: mkdir /tmp/bin; mkdir /tmp/lib; cp /usr/bin/python /tmp/bin; cp -r /usr/lib/pythonX.Y/ /tmp/lib/pythonX.Y; /tmp/bin/python. Then type import sys; sys.prefix. This will show "/tmp". If you remove the /tmp/lib directory, it will revert to "/usr". – Electrophoresis

Step 3 is the key thing I was missing--that the python binary looks in it's executable's neighborhood for associated files. Thanks. – Photography