venv not respecting --copies argument

Question

I am ssh’d into a development environment (vagrant Ubuntu box) and my project directory is mapped to another filesystem (via vbox) so symlinks are not supported. I am attempting to create a new venv, but the --copies flag isn’t being respected.

$sudo python -m venv --copies venv 
Error: [Errno 71] Protocol error: 'lib' -> '/home/vagrant/vagrant_projects/rurp/venv/lib64'

If I use python 2.7 ($virtualenv venv --always-copy) it works, but not with the python3 venv --copies implementation. The --always-copy argument was a workaround for similar issues with python2.x.

I could not find anything online indicating a bug in venv and am at a bit of a loss. Has anyone else had this issue?

$ python -V
Python 3.6.9

Thank you in advance.

Edit: Also tested in python 3.8.1.

Manamanacle

Looking at the source code for venv, --copies looks like a no-op unless you are using Windows. – Stupefacient

It set a defualt, but shouldn't that be overridden with the group.add_argument a few lines later? – Manamanacle

--copies uses store_false as its action, which just stores False in the destination, regardless of what is already there, so it's a no-op if the default is already False. – Stupefacient

Thanks! Do you think that this qualifies as a bug or just a documentation issue? – Manamanacle

Answer 1

Per @chepner's comment above, it looks like the --copies argument is ignored on non-Windows systems (no mention of this in the documentation). I was able to workaround the issue by creating the venv in a local directory, manually copying the symlinked lib64 to a real directory, moving the venv to my project folder and manually updating the activation scripts. Ugly, but it works.

$cd ~
$python3 -m venv --copies --clear venv
$cp -r --remove-destination `readlink lib64` lib64
$cp -r venv vagrant_project/rurp/

I would be happy to accept a more elegant answer.

Manamanacle

Answer 2

I ran into this problem on a Windows host and an Ubuntu box. No need for ugly workarounds. This github issue has the solution. Set the environment variable:

VIRTUALENV_ALWAYS_COPY=1 virtualenv /path/to/env

Milburr

The virtualenv package has a working --copies argument; only the built-in venv doesn't. – Keystone

Answer 3

The link posted by @chepner points to the default behaviour by OS.

• NT (windows): do not use symlinks (copy)
• otherwise (mac, linux, etc): use symlinks (don't copy)

        if os.name == 'nt':
            use_symlinks = False
        else:
            use_symlinks = True

use_symlinks is used by both of the mutually exclusive arguments --symlinks and --copies as a default option value (ie when the option is not specified on the CLI). Both these options store their value to the options object as symlinks with the appropriate truthiness depending on which option was used.

You can see the logic to copy or symlink depending on OS and this symlinks CLI option value is here.

Point is. Copies does work on non-windows systems. Something else is going on. It is possible you have a python binary in your venv directory that was copied into place from an incompatible system or that there is some soft-linking in the binary and the things that binary points to don't exist on the current system.

First thing to check is whether you have copies or symlinks in your virtual directory (e.g. ls -l venv/bin). If you have copies and not symlinks, try running the pythons: venv/bin/python3 and see if you still get an error. If you get an error it is an issue with the binary (not compatible with OS/arch or missing soft-linked libraries).

Copying around venv directories can have a bunch of problems. If you want to do this in a build/CI process make sure you really have the same os/system/arch on the CI as the destination machine and make sure you are working with copies and not symlinks. For activation and running python, make sure you update your PATH to include the venv python first export PATH="${PWD}/venv/bin:$PATH"

Statue