completely self-contained virtual environment

Question

I create a python3 virtual environment (explicitly avoiding symlinks, with --copies):

» python3 -m venv --without-pip --copies venv

This is my complete virtual environment now:

» tree venv/
venv/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── python
│   └── python3
├── include
├── lib
│   └── python3.4
│       └── site-packages
├── lib64 -> lib
└── pyvenv.cfg

I disable the PYTHONPATH, to make sure nothing is leaking from outside:

» PYTHONPATH=""

Activate the venv:

» source venv/bin/activate

Verify that activate has not polluted my PYTHONPATH:

» echo $PYTHONPATH

(blank, as expected)

I am using the right python:

» which python
/foo/bar/venv/bin/python

But the system modules are still being accessed:

» python 
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import unittest
>>> print(unittest)
<module 'unittest' from '/usr/lib/python3.4/unittest/__init__.py'>
>>> 

I would expect the import unittest statement to fail, since the virtual environment has no such module.

I would like to know:

• Why are system packages accessed when in a virtualenv?
• How can I create a completely self-contained virtual environment?

Musing

Since python 3.11 the venv docs explicitly say: "Not considered as movable or copyable – you just recreate the same environment in the target location." – Doormat

If it needs to be completely self-contained python environment, probably best to use portable python (or separate non-portable installation of python). Virtual environment are not meant to be completely isolated from the python installation, but to isolate the installed libraries ("site-packages"). – Hube

Answer 1

If I recall correctly the core system packages are symlinked, so they are the same files (partly to keep the size of the virtualenv down).

The default is not to include the site-packages directory, so it won't access 3rd party libraries that have been installed.

If you want to truly isolated and self-contained virtual environment, you might be better off looking at docker.

Virtualenv is really more of a lightweight way of managing different 3rd party installed packages for different apps.

EDIT:

It looks like --always-copy doesn't actually always copy all files:

virtualenv doesn't copy all .py files from the lib/python directory

Digging into the source and it looks like there's a smallish set of modules that are deemed to be "required" and these are the ones that are copied:

https://github.com/pypa/virtualenv/blob/ac4ea65b14270caeac56b1e1e64c56928037ebe2/virtualenv.py#L116

Edit 2:

You can see that the old python directories still appear in the sys.path, but after the directories for the virtualenv itself:

>>> import sys
>>> sys.path
['', '/home/john/venv/lib/python2.7', 
'/home/john/venv/lib/python2.7/plat-linux2', 
'/home/john/venv/lib/python2.7/lib-tk',
'/home/john/venv/lib/python2.7/lib-old', 
'/home/john/venv/lib/python2.7/lib-dynload', '/usr/lib/python2.7',
'/usr/lib/python2.7/plat-linux2',
'/usr/lib/python2.7/lib-tk',
'/home/john/venv/local/lib/python2.7/site-packages',
'/home/john/venv/lib/python2.7/site-packages']

Microstructure

Please note that I am using --copies: no symlinks are used (as seen in the directory tree). Docker is overkill for my needs: I can live with this situation, but I want to understand the background issues. Maybe a built-in PYTHONPATH in the system's python binary (which gets copied to the virtualenv)? How is this done? The system python binary is installed (in this case) with apt get, so how would the python binary know where it is installed (specially after being copied around!)? – Musing

Hadn't spotted the --copies there. I tested out using --always-copy locally (is --copies the older name?) and that does copy the files, but strangely not for unittest. If you try the same test for the os module, you should see that gives you the right path. – Microstructure

I do not have the --always-copy option, but --copies. I do not know which one is older. I am using python 3.4.3, and you? – Musing

And thanks for your explanation. It clarifies why are certain packages not copied with virtualenv (for me nothing is copied, actually, who knows why), but I still would like to know how can the python in the virtualenv use system packages (like unittest) – Musing

I was actually trying this with virtualenv 13.1.2 (on linux), but the docs only seem to refer to --always-copy. Odd. – Microstructure

The old python is still referenced on the PYTHONPATH - possible hardcoded in a binary. – Microstructure

Ok, got it. As you say, probably embedded in the python binary during compilation. As mentioned, my python is installed with apt, so using pre-compiled packages. It seems odd that such "installation-phase" info is hardcoded in a binary, though: it makes the installation directory for the binary not really relocatable (so basically, it can only be correctly installed in the path hardcoded during compilation time) – Musing

You can use --relocatable to make the virtualenv movable. It's quite common for binaries to have a hardcoded library path though, as it means you can multiple versions installed with different paths encoded in them. Everything then just works as expected. – Microstructure

Thanks, I know about --relocatable, but that is another matter altogether. What annoys me is that the python binary itself (not the virtualenv), has a hardcoded dependency to an installation path which is none of its business. I am the one to install it wherever I see fit (or apt in case of system python, or virtualenv in case of the virtualenv's python). – Musing

I would be less annoyed with python having a hardcoded relative reference to a lib path, but not with an absolute path. I assume these are linux/unix conventions, and there is surely a rationale of why this is the case (my last linux from scratch installation was done some time ago), but it is still surprising. – Musing

This is almost ten years old, but still, some disambiguation would be useful: it appears OP is asking about venv from the Python standard library, whereas this answer refers to pypa/virtualenv. Is that correct? I think this would also explain the --copies vs --always-copy confusion above. – Doormat

Answer 2

you can build python from source on an old Linux; then the python will be complete self-contained and able to run on many Linux.

in the reference link, it is the step detail to build python 3.6 on centos 5 in container env. the output can be used in many Linux like debian 10, centos 6, 7, 8, photon 3, alpine, ...

• readline-devel: enable python command line history
• sqlite-devel: enable sqlite support
• expat-devel: enable xmltree support
• bzip2-devel: enable bzip2 support
• compile SSL from source to avoid dynamic lib link
• if higher version of python, maybe you also need to compile higher version of gcc to get sth like full C++11 and higher support
• after compiling, you need to download setuptools and pip from pypi.python.org and do python setup.py install to get pip ready for this standalone python.

ref: https://github.com/stallpool/track-network-traffic#mitmproxy-for-manylinux

Common

Answer 3

Reading both answers, let me give mine:

• NO a venv is not standalone.

Its goal is just to isolate your libs from the system wide install of python.

Conda-pack propose what you are looking for. But it's an additional layer of software again. I think the best solution is to install a full isolated python.

Hibbard