What is the relationship between virtualenv and pyenv?

Question

I recently learned how to use virtualenv and virtualenvwrapper in my workflow but I've seen pyenv mentioned in a few guides but I can't seem to get an understanding of what pyenv is and how it is different/similar to virtualenv. Is pyenv a better/newer replacement for virtualenv or a complimentary tool? If the latter what does it do differently and how do the two (and virtualenvwrapper if applicable) work together?

Selfdelusion

fgimian.github.io/blog/2014/04/20/…: However, there is another alternative called pyenv which has several significant advantages. – Myriammyriameter

Don't be like me and get pyenv confused with pyvenv – Godrich

Answer 1

Pyenv and virtualenv are very different tools that work in different ways to do different things:

• Pyenv is a bash extension - will not work on Windows - that intercepts your calls to python, pip, etc., to direct them to one of several of the system python tool-chains. So you always have all the libraries that you have installed in the selected python version available - as such it is good for users who have to switch between different versions of python.

• VirtualEnv, is pure python so works everywhere, it makes a copy of, optionally a specific version of, python and pip local to the activate environment which may or may not include links to the current system tool-chain, if it does not you can install just a known subset of libraries into that environment. As such it is almost certainly much better for testing and deployment as you know exactly which libraries, at which versions, are used and a global change will not impact your module.

venv python > 3.3

Note that from Python 3.3 onward there is a built in implementation of VirtualEnv called venv (with, on some installations a wrapper called pyvenv - this wrapper is deprecated in Python 3.6), which should probably be used in preference. To avoid possible issues with the wrapper it is often a good idea to use it directly by using /path/to/python3 -m venv desired/env/path or you can use the excellent py python selector on windows with py -3 -m venv desired/env/path. It will create the directory specified with desired/env/path configure and populate it appropriately. In general it is very much like using VirtualEnv.

Additional Tools

There are a number of tools that it is worth mentioning, and considering, as they can help with the use of one or more of the above:

• VirtualEnvWrapper Manage and simplify the use and management of VirtualEnv - Cross Platform.
• pyenv-virtualenv, installed by pyenv-installer, which gives PyEnv tools for managing and interfacing to VirtualEnv - with this you can have a base installation that includes more than one version of python and create isolated environments within each of them - Linux/OS-X. Suggested by Johann Visagie
• PyInstaller can take your python code, possibly developed & tested under VirtualEnv, and bundle it up so that it can run one platforms that do not have your version of python installed - Note that it is not a cross compiler you will need a Windows (virtual-)machine to build Windows installs, etc., but it can be handy even where you can be sure that python will be installed but cannot be sure that the version of python and all the libraries will be compatible with your code.

Puppis

It probably worth mentioning pyenv-virtualenv, a pyenv plugin that allows pyenv and virtualenv to work seamlessly in unison. If you’ve installed pyenv using pyenv-installer, pyenv-virtualenv will already be present. * github.com/yyuu/pyenv-virtualenv * github.com/yyuu/pyenv-installer – Chatelaine

What about venv, now in the standard library? – Fearsome

@Flimm: From python 3.3 onward venv is a part of the standard library and should be used instead of virtualenv as it is often described as "virtualenv done right" - reddit.com/r/learnpython/comments/4hsudz/pyvenv_vs_virtualenv – Puppis

pyenv-virtualenv is not (or at least is no longer) linux only. There is a homebrew package for it for an easy install on MacOS. – Lebna

In recent days we developed pyenv for windows called pyenv-win. Have a look if it helps – Kaph

Answer 2

Short version:

• virtualenv allows you to create local (per-directory), independent python installations by cloning from existing ones
• pyenv allows you to install (build from source) different versions of Python alongside each other; you can then clone them with virtualenv or use pyenv to select which one to run at any given time

Longer version:

Virtualenv allows you to create a custom Python installation e.g. in a subdirectory of your project. This is done by cloning from an existing Python installation somewhere on your system (some files are copied, some are reused/shared to save space). Each of your projects can thus have their own python (or even several) under their respective virtualenv. It is perfectly fine for some/all virtualenvs to even have the same version of python (e.g. 3.8.5) without conflict - they live separately and don't know about each other. If you want to use any of those pythons from shell, you have to activate it (by running a script which will temporarily modify your PATH to ensure that that virtualenv's bin/ directory comes first). From that point, calling python (or pip etc.) will invoke that virtualenv's version until you deactivate it (which restores the PATH). It is also possible to call into a virtualenv Python using its absolute path - this can be useful e.g. when invoking Python from a script.

Pyenv operates on a wider scale than virtualenv. It is used to install (build from source) arbitrary versions of Python (it holds a register of available versions). By default, they're all installed alongside each other under ~/.pyenv, so they're "more global" than virtualenv. Then, it allows you to configure which version of Python to run when you use the python command (without virtualenv). This can be done at a global level or, separately, per directory (by placing a .python-version file in a directory). It's done by prepending pyenv's shim python script to your PATH (permanently, unlike in virtualenv) which then decides which "real" python to invoke. You can even configure pyenv to call into one of your virtualenv pythons (by using the pyenv-virtualenv plugin). You can also duplicate Python versions (by giving them different names) and let them diverge.

Using pyenv can be a convenient way of installing Python for subsequent virtualenv use.

Caricature

Answer 3

I'm frustrated enough by the lack of good explanations for this question for newcomers to post my own answer here, even though I'm not fully certain this is totally accurate and complete. I think that's because this space has had a lot of alternatives over the years and the "recommended" tools have changed so much.

Fundamental Difference: Python binary Vs. Python libraries

At fundamental level, when you want to start running any particular Python project, you need two things: the Python binary (and its built-ins), and some Python third party libraries; but you don't only have one project: that's why you need tools.

Clear Definitions: "Python" Vs. "Environment"

1. "Python". People often say "Python" when they mean more than just the binary, they include the libraries and third party installs as well. To run Python you need the actual binary; a single file, at a location, such as /usr/bin/python3. This also comes with the Python built-in libraries for the same version. These will be located in a directory somewhere like /usr/lib/python3/dist-packages. And it's also necessary to include the pip install in with the binary, as you can't get much further without pip to install your third party libraries.

2. "Environment". Again, the term "environment" is used so loosely no-one knows where it starts or ends or what particular kind of environment is being referred to. As soon as you start building your project one of the first things you're going to do is use pip to download and install third party libraries. These are installed into your local site_packages directory, which has its expected canonical locations, but can be anywhere as long as it's referenced properly so "Python" can find it. The site_packages directory is essentially your "environment".

Requirements: Multiple "Pythons" used with multiple "environments" each

So this is how it falls out.

Which "Python" is it?

You're going to start using more than one "Python". You'll have Python 3.7, Python 3.10, and it can grow. They can be re-used, but you need to install them all, which requires different locations, and then inform the tools in your project which path it is you are using for this project.

Which "Environment" is it?

And then even for one particular Python version, more than one project will use that same "Python", but with separate, different, "environments". One Python 3.10 project may use one version of the requests library, another 3.10 project may need a different version of requests library, and you don't want the installation of one to clobber the installation of the other: you need separate site_packages directories for each project.

Tools: "Python" management tool Vs. "Environment" management tool

This is where I lose some accuracy. I'm not familiar with all the tools, and they all have "env" in their name. Clever, huh? We can understand if they all have "py" in their name; that seems unavoidable. But if a tool is managing the Python installs but not the environment why is it called pyenv?

"Python" management tool

To be honest I haven't used pyenv before. I've managed Python installs just by using the OS package management, or I've just stuck to one version of Python. That came unstuck recently though, as systems can be dependent on a particular version of Python being in a particular place, so I want to use pyenv this time.

Maybe pyenv manages "environments" to some extent as well, but I'm using it to install multiple "Pythons". When I use pyenv (how to do that is out of scope for this answer) I want to install a particular version of Python, with its built-ins, and an associated pip.

There are few operations required for this because I only do this when I need a version of Python I haven't got yet. When I install Python 3.7.9 with pyenv, and then activate it, I can now start that Python version and use it. I can refer to it on the command line as python. I can say python -m pip install ....

But I haven't installed any libraries into my environment yet. Because I will use Python 3.7.9 on multiple projects, each with different environments...

"Environment" management tool

And that's where an environment management tool comes in. This question refers to virtualenv and virtualenvwrapper, but they're a bit outdated now, and this answer is focusing on the differences and reasons for the tools. I now use venv for creating and activating environments. Once I have selected (activate) a particular python I use that Python to create an "environment" that can also be activated. Once I have activated that environment, whenever I use python it will use the version of Python and (pip) that I created the environment with, and whenever I say pip install I will use that particular set of site_packages to install the third party libraries in.

Project Workflow

So this now sets you up to run multiple projects per Python, with multiple Pythons to choose from.

1. Create and / or activate the version of Python that you need, using your "Python management" tool: pyenv.
2. Using the particular activated Python, create and / or activate the environment for the project, using your "Environment management" tool: venv (or virtualenvwrapper).
3. Now every time you run Python it's the same version, and all pip installs reside in this local project only, and do not impact others.
4. And you can turn it off at any time (deactivate) and activate another one instead.
5. And you can have multiple shell sessions on multiple projects at the same time. Each Python and each environment is activated and referenced using the $PATH and other Linux environment variables in that shell session. Your shell prompt usually displays a name for you to be able to see which particular environment is currently activated.

Further Information

If you want more information about venv particularly, which (correct me if I'm wrong) seems to be the de facto standard at current Python versions you can find in-depth explanations on venv.

Antwanantwerp

I don't think "environment" is loose or ambiguous. The things that need to be there in order for the Python binary to work as intended. – Graf

One of the important features of virtual environments is that they are disposable. If you play around with different dependencies and install them as you go along, you will very soon end up in a situation where you don't know what you installed when, or how to recreate or document the situation. Being able to flush everything and start over makes this repeatable and reproducible. – Graf

This also has implications for overall system health; in the bad old days, you would have random versions of random libraries installed system-wide from whenever you needed them, but no good idea of what you had installed or what the implications might be for new projects or redeployments of old projects. Isolating the dependencies for each project or task removes this tendency to accumulate system-wide cruft. – Graf

very loose @tripleee, especially for noobs; and when you're focused on something else. It's so loose it's practically meaningless. I've tried to use a very clear demarcated meaning here, and you may also have a firm idea of what you mean by it in this context, but as a noob. a term like "environment", which can mean the OS, the IDE, Python as opposed to Ruby, the built-ins, third party libraries, a toolset like Bash, cloud as opposed to local, a deploy method, or any combination of the above, just saying the word "environment" immediately adds ambiguity. Thanks for commenting. – Antwanantwerp

pyenv does indeed allow - even require - you to create environments, but they are typically used across projects. So you might have "Python 3.7 plus Flask plus Requests" and "Python 3.10 plus Flask plus Requests" and switch between them in multiple projects if you are a web app developer. You could still overlay this with a regular virtual environment for each project (there is a separate pyenv-virtualenv plug-in for this). – Graf

I may not be exact about specific tools, but I'm trying to rise above that and provide an anchor by pointing out you need a Python version, and you need an environment. This should provide concepts for programmers to hang the different confusion of tools on. Programmers can start to rely on which one is for choosing Python and which one is for setting up an environment, and where they overlap if that does happen. I would recommend you look at venv (python -m venv --prompt name .venv) as I think it's quite a bit younger (and "more recommended" now) than pyenv-virtualenv). – Antwanantwerp

I use both, but primarily the regular venv for my daily needs. pyenv-virtualenv is designed for a different use case than mine, basically. – Graf

My impression is that pyenv installs Pythons, and then I need something else to manage multiple environments with those Pythons? Isn't that why you need pyenv-virtualenv? So your environment tool is not pyenv, it's pyenv-virtualenv? I'm just trying to get this fundamental distinction in this answer because I think this is where all the confusion arises. Noobs don't even know why they need more than one. – Antwanantwerp

Indeed, the base pyenv is only for managing separate Python versions and their standard libraries. You could install libraries system-wide for the Python version you selected with pyenv; but you probably don't want to, and so pyenv-virtualenv adds a facility for creating separate virtual environments which can be used across different projects. Sorry for being imprecise in my earlier comment. – Graf

Thanks @tripleee, it's good to get that confirmed. I think once someone understands the difference between the Python binary, the dist-packages and site-packages, they can start understanding what the tools are for and how to use them. – Antwanantwerp

@Antwanantwerp I agree with your objection: it's quite simple, "pyenv" has been badly named. It should have been called "pyversions". Other than that, I don't experience that much confusion from the other answers, even though, as regards pyenv specifically, I am indeed a "newb". – Rattlebox