Can virtualenvs be made to fall back to user packages instead of system packages?

Question

I use Jedi for Python autocompletion in Emacs, but it's not a dependency of my code so I don't want to put it in my requirements.txt. (Other developers may not use a Jedi editor plugin, and it's certainly not needed when I deploy to Heroku.)

But Jedi must be available from my virtualenv in order to function, i.e. if I can't

import jedi

it doesn't work.

Is there a good way to install Jedi user-globally such that it is available in all of my virtualenvs?

I think what I want is to

1. install Jedi into ~/.local/lib/python2.7/site-packages/ with pip's --user flag, then to
2. create my virtualenv using the equivalent of the --system-site-packages flag, but for user packages instead of system packages.

My current workaround is to pip install jedi in each of my virtualenvs. Then when I add new dependencies I pip install foo, pip freeze > requirements.txt, then manually remove jedi and a few other things from the file before committing. Obviously, this is time-consuming and error-prone.

Does anybody have a better solution?

Harangue

Why don't you install it in global site-packages? It would also help to know which Jedi plugin for Emacs you're using. – Manque

@DaveHalter, I'm using emacs-jedi intalled via MELPA. I do not install Jedi globally and fall back to that because I don't want my virtualenvs to see any of the plethora of other global packages that I have installed... If they are self-contained I can be more confident that I have all of my dependencies listed in requirements.txt. – Harangue

Answer 1

When virtuenv activates, it changes several env variables, such as PATH, PYTHONHOME, PS1, and so on, to point desired python binary, library, etc. You can change the script to change PYTHONPATH to use your user site-packages, namely ~/.local/lib/python2.7/site-packages, and possibly your system site-packages. With this setting, pip will search for library in virtual env, and then failover to user/system site-packages. Note that normally activate script does not change PYTHONPATH at all.

That is, add the following lines in your virtual_env/bin/activate.

# in activate script

    # in deactivate function
    if [ -n "$_OLD_VIRTUAL_PYTHONPATH" ] ; then
        PYTHONPATH="$_OLD_VIRTUAL_PYTHONPATH"
        export PYTHONPATH
        unset _OLD_VIRTUAL_PYTHONPATH
    fi

# in activate section
if [ -n "$PYTHONPATH" ] ; then
    _OLD_VIRTUAL_PYTHONPATH="$PYTHONPATH"
    PYTHONPATH=$HOME/.local/lib/python2.7/site-packages:/usr/lib/python2.7/site-packages
fi

Tergum

If your proposed change is durable, could I suggest you make this change out in the source code? github.com/pypa/virtualenv/blob/master/virtualenv_embedded/… It doesn't have value to us, that at project-run-time we're always running venv before we activate, resetting those scripts. – Farman