How to install Python (dev) dependencies globally so that I don't have to reinstall them in every venv?

Question

There are a few Python dependencies that I would like to be available in every venv (virtual environment) that I create for each project. For example black, flake8 and pytest. Is that possible and if so, how to achieve that?

I'd like to install these three once under my main Python installation, instead I have to reinstall all of them in every venv that I create when I start a new project. This is specially annoying when using VSCode which throws popups complaining about "Linter flake8 is not installed" or "...black is not installed", etc. when you switch to a venv where you haven't installed these packages.

Ovipositor

Did you try installing them globally? You can then use --system-site-packages when creating a virtualenv – Astylar

As I know virtual environment act as sperate isolated environment. So if you use virtual environment, you need to install all the packages with in that virtual environment. You can use @Astylar suggestions while creating virtual environment. – Kovacev

@Astylar THAT IS what I was looking for! Thank you! I have installed packages 'globally' but was not aware of this cli option. This does the trick - example: python -m venv --system-site-packages .venv/dev – Ovipositor

My next question would be "Is this possible w/ poetry, too?" Seems not yet --> github.com/python-poetry/poetry/issues/1393 – Ovipositor

Answer 1

Let me answer my own question based on comment from @jonrsharpe.

Assuming you want to have black, flake8 and pytest available 'globally' or in other words you want to have these packages in every new venv that you create but don't want to repeat pip install black flake8 pytest each and every time. Here's what you can do:

1. install the packages once under your main Python version (that you'd like to use for your venvs. NOTE: you make have several Python versions installed.)
2. when creating a new venv use --system-site-packages option. For example:

python -m venv --system-site-packages .venv/dev

3. activate your venv, i.e. source .venv/dev/bin/activate and check w/ pip list that the packages are available

Ovipositor

Answer 2

One Possible solution would be,

• Deactivate virtual environment.
• Install all packages which you need to be available globally.
• Again activate virtual environment.

• make sure you enable inherit package from global

Note: Please refer to these SO threads as well for more information.

• How do I install a pip package globally instead of locally?
• What does sudo -H do?

Carpus

'inherit package from global' is probably the term that could be in the title of my question was well. And --system-site-packages option (as commented by @Astylar above) is what let's me achieve exactly that. – Ovipositor

Answer 3

Don't use sudo pip install

If you use sudo pip install black, it will pollute your global Python installation. You don't want to do that.

You also don't need to use --system-site-packages in your virtualenvs. You can use it for other reasons, but using it just so that black etc. can work is a bad idea.

Intermediate solution: Use pip install --user

Deactivate your virtualenv, then execute pip install --user black (--user is the default in many systems when pip isn't in a venv/virtualenv and isn't run as root). This will install black somewhere in the user's home directory, such as $HOME/.local.

You will subsequently be able to run black regardless which venv/virtualenv you have activated, and regardless whether the venv/virtualenv has been created with --system-site-packages.

Why this works: If you wanted to import black in your code, this wouldn't work in a venv/virtualenv (unless with --system-site-packages), because black is installed "system-wide", albeit in a user's home dir ("user-wide" would be a more correct term in this case). However you don't want to import black; what you want is to execute the black command, and this will work regardless which venv/virtualenv you have activated, because to your shell black is just a command it can find in the path, just like ls or more (it's in $HOME/.local/bin, which should be in the PATH). This is true of pretty much everything you want to install "system-wide" (i.e. outside of any venv/virtualenv): shell commands like black, flake8 and pytest.

If you look at $HOME/.local/bin/black, you'll see that its first line is something like #!/usr/bin/python3. This means it uses the system-wide python installation and not a venv/virtualenv. (If it was #!/usr/bin/env python then it would use the activated venv/virtualenv instead.)

Best solution: Use pipx

The problem with the above solution is that it pollutes your "user-wide" python installation, which is almost like polluting your system-wide installation. You can do this instead (with venvs/virtualenvs deactivated):

pip install --user pipx
pipx install black

What pipx does is it creates a venv specifically for black and installs black in that venv. The $HOME/.local/bin/black executable that it creates is such that it runs black in that venv. pipx uninstall black removes both the executable and the venv.

Further reading

If there's still something unclear, my article virtualenv demystified may help.

Eutrophic