Broken references in Virtualenvs

Question

I recently installed a bunch of dotfiles on my Mac along with some other applications (I changed to iTerm instead of Terminal, and Sublime as my default text editor) but ever since, all my virtual environments have stopped working, although their folders inside .virtualenvs are still there and they give the following error whenever I try to run anything in them:

dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/[user]/.virtualenvs/modclass/bin/python
  Reason: image not found
Trace/BPT trap: 5

I have removed all the files related to dotfiles and have restored my .bash_profile to what it was before, but the problem persists. Is there any way to diagnose the problem or solve it in an easy way (e.g. not requiring to create all the virtualenvs all over again)?

Pahoehoe

Possibly related: debugfix.com/2011/11/dyld-library-loaded-executable_path-python – Perrine

Thank you for the comment, @unubtu. This certainly is helpful. But I am also not able to make any new virtualenvs. My rmvirtualenv still works but when trying to run mkvirtualenv, I get the following error: -bash: /usr/local/bin/virtualenv: /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/Resour: bad interpreter: No such file or directory So, it seems a problem with my python paths but I can't see where the problem is, since I can run python and it seems fine. – Pahoehoe

[update] I may have found the problem but I am not sure and I am actually not sure how to fix it. It seems that all virtualenv commands are working now in theory, but since there is a problem with python, they don't do anything. So the real problem is with brew's python. And I have a suspicion that the reason is because of a name change in python directories. For some reason, all these commands are looking for python in folder /usr/local/Cellar/python/2.7.6 but the folder's name is actually /usr/local/Cellar/python/2.7.6_1. – Pahoehoe

Since I am a novice, I don't know how risky it is to manually change the name from 2.7.6_1 to 2.7.6 and see what happens. – Pahoehoe

You should be able to rename 2.7.6_1 to 2.7.6. If worse came to worst, you could rename it back. – Perrine

Yup. I actually went ahead and tried it. That actually made me notice another problem. The problem wasn't the name, but rather it seemed it was broken links, somehow caused in brew. I tried a bunch of things and the problem seems to be gone now. What seems to have worked was uninstalling Python in homebrew all together and installing it again and then running brew unlink python && brew link python followed by brew linkapps. Although I think the second part wasn't even necessary since I had tried it before to no avail. Thanks again for the help! – Pahoehoe

Broken links didn't work for me, unfortunately. I was able to work around the 'bad interpreter' problem by @oxtay's solution, by making a new directory (mkdir) called /usr/local/Cellar/python/2.7.6 and copying (cp -r) all of the files from /usr/local/Cellar/python/2.7.9 into that folder. I was too nervous to just rename the folder! – Ruthenium

Answer 1

I found the solution to the problem here, so all credit goes to the author.

The gist is that when you create a virtualenv, many symlinks are created to the Homebrew installed Python.

Here is one example:

$ ls -la ~/.virtualenvs/my-virtual-env
...
lrwxr-xr-x  1 ryan staff   78 Jun 25 13:21 .Python -> /usr/local/Cellar/python/2.7.7/Frameworks/Python.framework/Versions/2.7/Python
...

When you upgrade Python using Homebrew and then run brew cleanup, the symlinks in the virtualenv point to paths that no longer exist (because Homebrew deleted them).

The symlinks needs to point to the newly installed Python:

lrwxr-xr-x  1 ryan staff   78 Jun 25 13:21 .Python -> /usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/Python

The solution is to remove the symlinks in the virtualenv and then recreate them:

find ~/.virtualenvs/my-virtual-env/ -type l -delete
virtualenv ~/.virtualenvs/my-virtual-env

It's probably best to check what links will be deleted first before deleting them:

find ~/.virtualenvs/my-virtual-env/ -type l

In my opinion, it's even better to only delete broken symlinks. You can do this using GNU find:

gfind ~/.virtualenvs/my-virtual-env/ -type l -xtype l -delete

You can install GNU find with Homebrew if you don't already have it:

brew install findutils

Notice that by default, GNU programs installed with Homebrew tend to be prefixed with the letter g. This is to avoid shadowing the find binary that ships with OS X.

Interfertile

+1 gfind was perfect, since I had a lot of unbroken symlinks (e.g., nodeenv) that I didn't want to delete – Leandra

Another way to remove broken symlinks is using the standard find: find -L ~/.virtualenvs/my-virtual-env/ -type l | xargs rm – Walden

I deleted my entire virtualenv dir. now i cannot remove symlinks. Non of the solutions mentioned on this page work for me on mac. i still get same error "image not found . Abort trap: 6" – Jueta

These steps didn't quite work for me: pip3 freeze dyld: lazy symbol binding failed: Symbol not found: __Py_UnixMain – Presswork

Just to add, if the env was with Python 2, run it with argument: virtualenv ~/.virtualenvs/foo -p python2, otherwise it will use Python 3. – Capriole

Thank you very much. Deleting the symlinks and recreating the virtualenv did the trick. Well done – Elfin

Thank you so much for this answer! Another very hacky solution that I tried was to unlink brew from the upgraded version of Python, and reinstalling the previous version. – Giglio

This didn't work for me, but Robert Brisita's suggestion of reinstalling virtualenvwrapper did – Sylvanus

Answer 2

After trying a few things, this worked for me:

go to your virtualenv directory (but don't run workon):

cd ~/.virtualenv/name_of_broken_venv

Now delete these files:

rm -rf .Python bin/python* lib/python2.7/* include/python2.7

Then to rebuild your venv, run:

virtualenv .
workon name_of_broken_venv
pip freeze

You should now see a list of your installed packages again.

Hulburt

FWIW, I just tried this approach after upgrading to El Capitan and re-installing homebrew, and my package list was not preserved. – Isoprene

with pipenv you can remove by doing pipenv --rm and recreate, pipenv shell,pipenv install – Fauver

If you get errors related to virtualenvwrapper.sh: There was a problem running the initialization hooks you will need to pip install virtualenv virtualenvwrapper or pip3 install virtualenv virtualenvwrapper just to get everything back in sync. – Carrelli

Answer 3

This occurred when I updated to Mac OS X Mavericks from Snow Leopard. I had to re-install brew beforehand too. Hopefully you ran the freeze command for your project with pip.

To resolve, you have to update the paths that the virtual environment points to.

• Install a version of python with brew:

brew install python

• Re-install virtualenvwrapper.

pip install --upgrade virtualenvwrapper

• Removed the old virtual environment:

rmvirtualenv old_project

• Create a new virtual environment:

mkvirtualenv new_project

• Work on new virtual environment

workon new_project

• Use pip to install the requirements for the new project.

pip install -r requirements.txt

This should leave the project as it was before.

Zilber

This was a while ago and I believe I eventually did something along these lines, but since I hadn't run 'pip freeze > requirements.txt' back then, it wasn't the most efficient solution. Lesson learned. – Pahoehoe

Answer 4

A update version @Chris Wedgwood's answer for keeping site-packages (keeping packages installed)

cd ~/.virtualenv/name_of_broken_venv


mv lib/python2.7/site-packages ./    
rm -rf .Python bin lib include
virtualenv .
rm -rf lib/python2.7/site-packages
mv ./site-packages lib/python2.7/

Mcneely

This is beyond perfection. Helps migrate python version while retaining all the packages. If you are following this, do not execute @Chris Wedgewood's instructions. – Lacielacing

Answer 5

It appears the proper way to resolve this issue is to run

 pip install --upgrade virtualenv

after you have upgraded python with Homebrew.

This should be a general procedure for any formula that installs something like python, which has it's own package management system. When you install brew install python, you install python and pip and easy_install and virtualenv and so on. So, if those tools can be self-updated, it's best to try to do so before looking to Homebrew as the source of problems.

Showiness

This worked for an issue with setuptools, specifically: Warning: cannot find svn location for setuptools==0.6c12dev-r88846 – Zilber

I applied this solution, followed by running: virtualenv . in my broken virtual environment. The updated version of virtualenv then recreated the necessary dependencies and I was good to go. This process was more self-managed and robust than the accepted answer for me. – Seismology

In 2020, this is still the answer. – Reimburse

Answer 6

If this was caused by a brew upgrade that upgraded its Python, and you're ok with downgrading to the previous version, try brew switch python [previous version], eg brew switch python 3.6.5. From here.

Cartwell

Answer 7

Anyone who is using pipenv (and you should!) can simply use these two commands — without having the venv activated:

rm -rf `pipenv --venv` # remove the broken venv
pipenv install --dev   # reinstall the venv from pipfile 

Espy

you also can use pipenv --rm in the folder of your env and then pipenv install --dev – Ahola

Answer 8

virtualenvwrapper instructions

As indicated in the accepted answer, the root cause is likely a homebrew update that means your virtualenv symlinks are pointing at broken python paths - see details here.

For each virtual env, you need to reassign the symlinks to point at the correct python path (in brew cellar). Here is how to do it with virtualenvwrapper. Here I am updating a virtual env called "my-example-env".

cd ~/PYTHON_ENVS
find ./my-example-env -type l -delete
mkvirtualenv my-example-env

All done.

Sultanate

Not portable (BSD find) – Spatola

Answer 9

If you've busted python3 just try brew upgrade python3 that fixed it for me.

Gruelling

Answer 10

I recently faced this. None of the above solutions worked for me. Seems it wasn't actually Python's problem. When I was running

aws s3 ls

I was getting following error:

dyld: Library not loaded: @executable_path/../.Python

This means, the library aws executable is pointing towards is either doesn't exist or is corrupted, thus I uninstalled and reinstalled aws-cli following instructions from this link and it worked!!

Bucaramanga

Answer 11

The problem for me(a MacOS user) is that brew updated the Python and virtualenvs links to the old version which was deleted.

We can check and fix it by

>> ls -al ~/.virtualenvs/<your-virtual-env>/.Python
.Python -> /usr/local/Cellar/python/<old-version>/Frameworks/Python.framework/Versions/3.7/Python
>> rm ~/.virtualenvs/<your-virtual-env>/.Python
>> ln -s  /usr/local/Cellar/python/<new-version>/Frameworks/Python.framework/Versions/3.7/Python ~/.virtualenvs/<your-virtual-env>/.Python

Orleanist

This also worked to fix broken links after installing Python 3.7 on a system that had Python3.6 – Saccharo

Answer 12

I had a similar issue and i solved it by just rebuilding the virtual environment with virtualenv .

Wrigley

Welcome to SO. Though we thank you for your answer, it would be better if it provided additional value on top of the other answers. In this case, your answer does not provide additional value, since another user already posted that solution. If a previous answer was helpful to you, you should vote it up once you have enough reputation – Irritating

Answer 13

Using Python 2.7.10.

A single command virtualenv path-to-env does it. documentation

$ virtualenv path-to-env
Overwriting path-to-env/lib/python2.7/orig-prefix.txt with new content
New python executable in path-to-env/bin/python2.7
Also creating executable in path-to-env/bin/python
Installing setuptools, pip, wheel...done.

Balbur

Answer 14

I had a broken virtual env due to a Homebrew reinstall of python (thereby broken symlinks) and also a few "sudo pip install"s I had done earlier. Weizhong's tips were very helpful in fixing the issues without having to reinstall packages. I also had to do the following for the mixed permissions problem.

sudo chown -R my_username lib/python2.7/site-packages

Frecklefaced

If you're complementing another user's answers you should leave a comment for them so they can edit! Nice contribution. – Crocidolite

He doesn't have enough reputation points to comment on an answer. – Fingertip

Answer 15

Virtualenvs are broken. Sometimes simple way is to delete venv folders and recreate virutalenvs.

Cainozoic

Answer 16

If you using pipenv, just doing pipenv --rm solves the problem.

Oilskin

Answer 17

The accepted answer does not work for me: the file $WORKON_HOME/*/bin/python2.7 is no longer a symlink, it is a full-fledged executable:

$ file $WORKON_HOME/*/bin/python2.7
/Users/sds/.virtualenvs/.../bin/python2.7: Mach-O 64-bit executable x86_64
...

The solution is, alas, to completely remove and re-create from scratch all the virtual environments.

For the reference:

deactivate
pip install --user virtualenv virtualenvwrapper
pip install --user --upgrade virtualenv virtualenvwrapper
for ve in $(lsvirtualenv -b); do
  # assume that each VE is associated with a project
  # and the project has the requirements.txt file
  project=$(cat $WORKON_HOME/$ve/.project)
  rmvirtualenv $ve
  mkvirtualenv -a $project -r requirements.txt $ve
done

Calcific

I guess it is because this solution is not obsolete -- I have just tried it and it fixed my issue. Also, I think if you don't have symlinks, you won't see the error described here, so this comment becomes not a solution but a distraction -- Just because you have a newer version, doesn't mean everybody does. That's my guess why the downvote :) – Passus

@RafazZ: I hope it is now better. However, I wonder why it is still a symlink for you. And yes, I do get that error because the virtualenv python is linked against the stock python libs. – Calcific

I think the default behaviour is still to create symlinks and you need an --always-copy argument to override it. At least that what I got from the User Guide – Passus

@RafazZ: I never used --always-copy and I have regular files :-( – Calcific

Answer 18

Simply upgrading python3 worked for me:

brew upgrade python3

Marjoriemarjory

Answer 19

What fixed it for me was just uninstalling python3 and pipenv then reinstalling them.

brew uninstall pipenv
brew uninstall python3
brew install python3 
brew install pipenv

Industry

Answer 20

I tried the top few methods, but they didn't work, for me, which were trying to make tox work. What eventually worked was:

sudo pip install tox

even if tox was already installed. The output terminated with:

Successfully built filelock
Installing collected packages: py, pluggy, toml, filelock, tox
Successfully installed filelock-3.0.10 pluggy-0.11.0 py-1.8.0 toml-0.10.0 tox-3.9.0

Aer

Answer 21

All the answers are great here, I tried a couple of solutions mentioned above by Ryan, Chris and couldn't resolve the issue, so had to follow a quick and dirty way.

1. rm -rf <project dir> (or mv <project dir> <backup projct dir> if you want to keep a backup)
2. git clone <project git url>
3. Move on!

Nothing novel here, but it makes life easier!

Kneepad

Answer 22

I am sure I am late to the party but I want to say that the resolution of this problem is much simpler than discussed here.

You can easily regenerate the virtual environment without having to delete/edit anything. Assuming that your broken environment is called env_to_fix you can just to the following:

mkvirtualenv env_to_fix

This will regenerate the links and fix the environment without the need to dump the current status somewhere and restore it.

Jessikajessup

Answer 23

I came across the same issue when I was pointing my python run time from 2 to 3 on my mac, pointing the alias python to python 3 path. I then recreate a new virtualenv and re-install those packages i need for my project. For my use case i have had a python program writing to google sheet. Clean up a few packages that are different from python 2 implementation and wa la, things started working again.

Mesh

Answer 24

I was facing the same issue after upgrading brew on my OSX Catalina.

After trying bunch of stuffs, I find the following is the best and easy solution.

At first, delete the virtual env. (Optional)

find myvirtualenv -type l -delete

then recreate a new virtualenv

virtualenv myvirtualenv

Reference: https://www.jeremycade.com/python/osx/homebrew/2015/03/02/fixing-virtualenv-after-a-python-upgrade/

Ezana

Answer 25

So there are many ways but one which worked for me is as follows since I already had my requirements.txt file freeze.

So delete old virtual environment with following command use deactivate cd .. rm -r old_virtual_environment

to install virtualenv python package with pip use pip install virtualenv

then check if it's installed correctly use virtualenv --version

jump to your project directory use cd project_directory

now create new virtual environment inside project directory using following use virtualenv name_of_new_virtual_environment

now activate newly created virtual environment use source name_of_new_virtual_environment/bin/activate

now install all project dependencies using following command use pip install -r requirements.txt

Cobra

Answer 26

When you are running into this issue on a freshly created virtualenv, it might be that your python version installed by brew is "unlinked".

You can fix this for example by running: brew link [email protected] (but specify your speficic python version)

You can also run brew doctor, it will tell you if you have unlinked stuff and how to fix this.

Biestings