Upgrade Python in a virtual environment

Question

Is there a way to upgrade the version of Python used in a virtual environment (e.g. if a bugfix release comes out)?

I could pip freeze --local > requirements.txt, then remove the directory and pip install -r requirements.txt, but this requires a lot of reinstallation of large libraries, for instance, numpy, which I use a lot.

I can see this is an advantage when upgrading from, e.g., 2.6 -> 2.7, but what about 2.7.x -> 2.7.y?

Knapweed

While you may be able to modify some paths, etc. to get it to work. The last thing you want is a slightly different environment from production. Just think.. if there is even one difference between how the various versions compile, you're going to loose out on all the time saved by tracking down the bug. I'd just take the time to create a brand new virtualenv and reinstall everything. – Lorimer

@Simon Walker. Exactly same problem. I have to upgrade python 3.5.3 for a bug fix. – Eatton

Answer 1

Did you see this? If I haven't misunderstand that answer, you may try to create a new virtualenv on top of the old one. You just need to know which python is going to use your virtualenv (you will need to see your virtualenv version).

If your virtualenv is installed with the same python version of the old one and upgrading your virtualenv package is not an option, you may want to read this in order to install a virtualenv with the python version you want.

EDIT

I've tested this approach (the one that create a new virtualenv on top of the old one) and it worked fine for me. I think you may have some problems if you change from python 2.6 to 2.7 or 2.7 to 3.x but if you just upgrade inside the same version (staying at 2.7 as you want) you shouldn't have any problem, as all the packages are held in the same folders for both python versions (2.7.x and 2.7.y packages are inside your_env/lib/python2.7/).

If you change your virtualenv python version, you will need to install all your packages again for that version (or just link the packages you need into the new version packages folder, i.e: your_env/lib/python_newversion/site-packages)

Siobhansion

He specifically did not want to reinstall any packages if possible. Obviously you have never installed numpy from source if you do not know why it matters ;) hint: it takes a LOOOOOOOOOOOOOONG time. – Duckett

yeah, i understand that. But it seems that if he make a virtualenv in top of the other, packages are not lost, so he won't have to reinstall numpy or any other package. Nevertheless, i think he should try this in a new virtualenv in case it fails. – Siobhansion

I rest my case, I believe that the new virtualenv versions do indeed upgrade in place, however for old versions, they did not even overwrite the python executable, which caused problems with at least one python bugfix release. – Duckett

It didn't work for me and looks like it's not supposed to: github.com/pypa/virtualenv/issues/437 – Clere

Can you show us the error you get @Kentzo? It probably won't work in all cases, but it surely works in some cases. Maybe if we show us the error you get we can understand it and help you... – Siobhansion

I tried to create a new virtualenv on top of the old one, and it did work. I did have to specify -p to point it to the right version of python. – Pennypennyaliner

Whatever you do, don't remove the old python version from your system before upgrading the virtualenv because otherwise you can't do the install over itself trick. – Crossgarnet

Answer 2

If you happen to be using the venv module that comes with Python 3.3+, it supports an --upgrade option. Per the docs:

Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place

python3 -m venv --upgrade ENV_DIR

Gitt

I upgraded python from 3.5.1 to 3.6.4 on my system. And to upgrade python on my virtualenv, I had to explicitly mention python3.6 in the command : python3.6 -m venv --upgrade <path_of_existing_env> – Defoliate

The --upgrade flag is not necessary in my experience (3.5.2 to 3.5.4 on Windows/Git Bash). – Nazler

In my case, bin/python and bin/python3 still linked to the old python3.4 binary, and I had to set the links manually. Also, is there a way to remove the now outdated python version? – Antilebanon

I got Error: Command '['/Users/me/Sites/site/venv3/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. > what does it mean ? I see no active python processes, cannot upgrade. – Eadie

@Eadie I got a similar error after I upgraded my system Python version from 3.5 to 3.6 (and removed 3.5). I had to reinstall 3.5 and run python3.6 -m venv --upgrade ENV_DIR for it to work. I think I'll still have to update some symbolic links in ENV_DIR/bin. Hence, the trick was to update with the new Python version while the old version was still installed. – Tangential

@Eadie for those getting the subprocess error called python -Im ensurepip to see what the actual underlying error is – Inflammable

@Eadie I had a similar error and found that adding the --without-pip flag solved the issue for me. – Ashlieashlin

@Anupam it worked for me for python 3.9.5 python3.9 -m venv --upgrade venv_dir/. – Astrograph

did not work for me: V3.6.8 -> V3.8.10. F:\MyCodes\python\dtprjops>python -m venv --upgrade venv Error: [Errno 13] Permission denied: 'F:\\MyCodes\\python\\dtprjops\\venv\\Scripts\\python.exe' – Stereochromy

@Antilebanon this package is very confusing but recreating the venv did it for me. Caveat: it seems you must invoke --clear with the Python version you intend to upgrade to and then, obviously, recreate it with the updated Python version. I cleared with the old version and things got weird. – Forbes

@Eadie for me the problem was a missing lsb_release. The new python version doesn't link to it, so I fixed it with sudo ln -s /usr/share/pyshared/lsb_release.py /usr/local/lib/python3.xx/site-packages/lsb_release.py – Muscadine

Using 3.9 -> 3.11, this worked for most things, but not numpy (and hence pandas etc). Perhaps it doesn't deal with C extensions cleanly. I had a requirements.txt already, and an extra ` pip install -r requirements.txt --force-reinstall` cleaned it up. – Slipstream

@Jens, AFAIK, venv doesn't add Python in the first place, so it doesn't remove it either. AFAIK venv copies the binary. An upgrade would overwrite that. Use of pyenv always involves use of an already extant Python external to venv; so it's not part of the role of venv to manage your Pythons for you. – Milliard

@Milliard I think that depends on how you set up your venv — use symlinks to the system’s Python binaries, or use copies. – Antilebanon

There are so many differences no wonder this is a confusing space @Jens. Pity there isn't just one obvious way to do it! But if your env Python is only linked there's even less need to remove it. Links will get deleted if you delete the environment. Links will get overwritten if you upgrade. At the end of the day the whole way of establishing a venv environment requires you to call an already extant Python before the environment is created, which forces it therefore to be external. There's no way venv should manage your Python for you. You can delete it afterwards if you want to. – Milliard

Answer 3

Step 1: Freeze requirement & take a back-up of existing env

pip freeze > requirements.txt
deactivate
mv env env_old

Step 2: Install Python 3.7 & activate virutal environment

sudo apt-get install python3.7-venv
python3.7 -m venv env
source env/bin/activate
python --version

Step 3: Install requirements

sudo apt-get install python3.7-dev
pip3 install -r requirements.txt

Serpigo

Answer 4

Updated again: The following method might not work in newer versions of virtualenv. Before you try to make modifications to the old virtualenv, you should save the dependencies in a requirement file (pip freeze > requirements.txt) and make a backup of it somewhere else. If anything goes wrong, you can still create a new virtualenv and install the old dependencies in it (pip install -r requirements.txt).

Updated: I changed the answer 5 months after I originally answered. The following method is more convenient and robust.

Side effect: it also fixes the Symbol not found: _SSLv2_method exception when you do import ssl in a virtual environment after upgrading Python to v2.7.8.

Notice: Currently, this is for Python 2.7.x only.

---

If you're using Homebrew Python on OS X, first deactivate all virtualenv, then upgrade Python:

brew update && brew upgrade python

Run the following commands (<EXISTING_ENV_PATH> is path of your virtual environment):

cd <EXISTING_ENV_PATH>
rm .Python
rm bin/pip{,2,2.7}
rm bin/python{,2,2.7}
rm -r include/python2.7
rm lib/python2.7/*
rm -r lib/python2.7/distutils
rm lib/python2.7/site-packages/easy_install.*
rm -r lib/python2.7/site-packages/pip
rm -r lib/python2.7/site-packages/pip-*.dist-info
rm -r lib/python2.7/site-packages/setuptools
rm -r lib/python2.7/site-packages/setuptools-*.dist-info

Finally, re-create your virtual environment:

virtualenv <EXISTING_ENV_PATH>

By doing so, old Python core files and standard libraries (plus setuptools and pip) are removed, while the custom libraries installed in site-packages are preserved and working, as soon as they are in pure Python. Binary libraries may or may not need to be reinstalled to function properly.

This worked for me on 5 virtual environments with Django installed.

BTW, if ./manage.py compilemessages is not working afterwards, try this:

brew install gettext && brew link gettext --force

Justen

The only file I had to delete was the bin/python file. – Orlina

For some older version Python, deleting setuptools and pip is necessary. – Justen

I also had to delete <EXISTING_ENV_PATH>/.Python as that broke the virtualenv creation. – Pulliam

Based on this answer I wrote a little script (with full attribution): burgundywall.com/post/update-all-python-virtual-environments – Equalitarian

You can actually easily remove your virtual environment by running rmvirtualenv <env_name> and will remove all its lined dependencies :) See more at: virtualenvwrapper.readthedocs.io/en/latest/… – Particularly

@Nick Lucas In fact, I wanted to keep all the lined dependencies, while removing the core Python files and standard libraries :) – Justen

Answer 5

How to upgrade the Python version for an existing virtualenvwrapper project and keep the same name

I'm adding an answer for anyone using Doug Hellmann's excellent virtualenvwrapper specifically since the existing answers didn't do it for me.

Some context:

• I work on some projects that are Python 2 and some that are Python 3; while I'd love to use python3 -m venv, it doesn't support Python 2 environments
• When I start a new project, I use mkproject which creates the virtual environment, creates an empty project directory, and cds into it
• I want to continue using virtualenvwrapper's workon command to activate any project irrespective of Python version

Directions:

Let's say your existing project is named foo and is currently running Python 2 (mkproject -p python2 foo), though the commands are the same whether upgrading from 2.x to 3.x, 3.6.0 to 3.6.1, etc. I'm also assuming you're currently inside the activated virtual environment.

1. Deactivate and remove the old virtual environment:

$ deactivate
$ rmvirtualenv foo

Note that if you've added any custom commands to the hooks (e.g., bin/postactivate) you'd need to save those before removing the environment.

2. Stash the real project in a temp directory:

$ cd ..
$ mv foo foo-tmp

3. Create the new virtual environment (and project dir) and activate:

$ mkproject -p python3 foo

4. Replace the empty generated project dir with real project, change back into project dir:

$ cd ..
$ mv -f foo-tmp foo
$ cdproject

5. Re-install dependencies, confirm new Python version, etc:

$ pip install -r requirements.txt
$ python --version

If this is a common use case, I'll consider opening a PR to add something like $ upgradevirtualenv / $ upgradeproject to virtualenvwrapper.

Jargon

Yes, please. I got halfway through this and got major problems... A check would be nice because the first step is irreversible once deleted (to my current knowledge) – Eldin

I found it necessary to also install distutils for my python version to ensure that the new mkvirtualenv command worked without errors: sudo apt-get install python3.10-distutils – Prescript

Answer 6

Let's consider that the environment that one wants to update has the name venv.

1. Backup venv requirements (optional)

First of all, backup the requirements of the virtual environment:

pip freeze > requirements.txt
deactivate
#Move the folder to a new one
mv venv venv_old

2. Install Python

Assuming that one doesn't have sudo access, pyenv is a reliable and fast way to install Python. For that, one should run

curl https://pyenv.run | bash

and then

exec $SHELL

as suggested here.

If, when one tries to update pyenv

pyenv update

and one gets the error

bash: pyenv: command not found

that is because pyenv path wasn't exported to .bashrc. It can be solved by executing the following commands:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc

Then restart the shell

exec "$SHELL"

Now one should install the Python version that one wants. Let's say version 3.8.3

pyenv install 3.8.3

One can confirm if it was properly installed by running

pyenv versions

The output should be the location and the versions (in this case 3.8.3)

3. Create the new virtual environment

Finally, with the new Python version installed, create a new virtual environment (let's call it venv)

python3.8 -m venv venv

Activate it

source venv/bin/activate

and install the requirements

pip install -r requirements.txt

Now one should be up and running with a new environment.

Bandit

Answer 7

I wasn't able to create a new virtualenv on top of the old one. But there are tools in pip which make it much faster to re-install requirements into a brand new venv. Pip can build each of the items in your requirements.txt into a wheel package, and store that in a local cache. When you create a new venv and run pip install in it, pip will automatically use the prebuilt wheels if it finds them. Wheels install much faster than running setup.py for each module.

My ~/.pip/pip.conf looks like this:

[global]
download-cache = /Users/me/.pip/download-cache
find-links =
/Users/me/.pip/wheels/

[wheel]
wheel-dir = /Users/me/.pip/wheels

I install wheel (pip install wheel), then run pip wheel -r requirements.txt. This stores the built wheels in the wheel-dir in my pip.conf.

From then on, any time I pip install any of these requirements, it installs them from the wheels, which is pretty quick.

Abruzzi

For most people I think making use of pip machinery to do the work makes sense, but for anyone who has customized their virtualenvs postactivate, predeactivate, etc, this is not a desirable option because all that is lost and has to be manually readded. – Crossgarnet

Answer 8

I just want to clarify, because some of the answers refer to venv and others refer to virtualenv.

Use of the -p or --python flag is supported on virtualenv, but not on venv. If you have more than one Python version and you want to specify which one to create the venv with, do it on the command line, like this:

malikarumi@Tetuoan2:~/Projects$ python3.6 -m venv {path to pre-existing dir you want venv in}

You can of course upgrade with venv as others have pointed out, but that assumes you have already upgraded the Python that was used to create that venv in the first place. You can't upgrade to a Python version you don't already have on your system somewhere, so make sure to get the version you want, first, then make all the venvs you want from it.

Torytoryism

Answer 9

This approach always works for me:

# First of all, delete all broken links. Replace  my_project_name` to your virtual env name
find ~/.virtualenvs/my_project_name/ -type l -delete
# Then create new links to the current Python version
virtualenv ~/.virtualenvs/my_project_name/
# It's it. Just repeat for each virtualenv located in ~/.virtualenvs

Taken from:

• https://github.com/1st/python-on-osx#python-virtualenv
• https://gist.github.com/1st/ced02a1c64ac7b82bb27e432eea6b068

Squirm

Answer 10

If you're using pipenv, I don't know if it's possible to upgrade an environment in place, but at least for minor version upgrades it seems to be smart enough not to rebuild packages from scratch when it creates a new environment. E.g., from 3.6.4 to 3.6.5:

$ pipenv --python 3.6.5 install
Virtualenv already exists!
Removing existing virtualenv…
Creating a v$ pipenv --python 3.6.5 install
Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…
Using /usr/local/bin/python3.6m (3.6.5) to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/python3.6
Also creating executable in /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD
Installing dependencies from Pipfile.lock (84dd0e)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 47/47 — 00:00:24
To activate this project's virtualenv, run the following:
 $ pipenv shell
$ pipenv shell
Spawning environment shell (/bin/bash). Use 'exit' to leave.
. /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/activate
bash-3.2$ . /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/activate
(autoscale-aBUhewiD) bash-3.2$ python
Python 3.6.5 (default, Mar 30 2018, 06:41:53) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>>

Refuse

You refer to this pipenv? – Antilebanon

Answer 11

I moved my home directory from one mac to another (Mountain Lion to Yosemite) and didn't realize about the broken virtualenv until I lost hold of the old laptop. I had the virtualenv point to Python 2.7 installed by brew and since Yosemite came with Python 2.7, I wanted to update my virtualenv to the system python. When I ran virtualenv on top of the existing directory, I was getting OSError: [Errno 17] File exists: '/Users/hdara/bin/python2.7/lib/python2.7/config' error. By trial and error, I worked around this issue by removing a few links and fixing up a few more manually. This is what I finally did (similar to what @Rockalite did, but simpler):

cd <virtualenv-root>
rm lib/python2.7/config
rm lib/python2.7/lib-dynload
rm include/python2.7
rm .Python
cd lib/python2.7
gfind . -type l -xtype l | while read f; do ln -s -f /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/${f#./} $f; done

After this, I was able to just run virtualenv on top of the existing directory.

Ildaile

Answer 12

On OS X or macOS using Homebrew to install and upgrade Python3 I had to delete symbolic links before python -m venv --upgrade ENV_DIR would work.

I saved the following in upgrade_python3.sh so I would remember how months from now when I need to do it again:

brew upgrade python3
find ~/.virtualenvs/ -type l -delete
find ~/.virtualenvs/ -type d -mindepth 1 -maxdepth 1 -exec python3 -m venv --upgrade "{}" \;

UPDATE: while this seemed to work well at first, when I ran py.test it gave an error. In the end I just re-created the environment from a requirements file.

Stentor

py.test doesn't work because `~/.virtualenvs/myenv/bin/python3.7 isn't a symlink, so doesn't get updated. – Pierce

Ah actually, the problem is There is ~/.virtualenvs/myenv/.Python which is a symlink. – Pierce

Answer 13

For everyone with the problem

Error: Command '['/Users/me/Sites/site/venv3/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

You have to install python3.6-venv

 sudo apt-get install python3.6-venv

Premise