On OS X El Capitan I can not upgrade a python package dependent on the six compatibility utilities NOR can I remove six

Question

I am trying to use scrape, but I have a problem.

from six.moves import xmlrpc_client as xmlrpclib

ImportError: cannot import name xmlrpc_client

Then, I tried pip install --upgrade six scrape, but:

  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 211, in main
status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 311, in run
root=options.root_path,
  File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 716, in uninstall
paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 315, in renames
shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)

OSError: [Errno 1] Operation not permitted: '/var/folders/3h/r_2cxlvd1sjgzfgs4xckc__c0000gn/T/pip-5h86J8-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

Irredeemable

Answer 1

Quick Fix:

I just got around what I think was the same problem. You might consider trying this (sudo, if necessary):

pip install scrape --upgrade --ignore-installed six

Github is ultimately where I got this answer (and there are a few more suggestions you may consider if this one doesn't solve your problem). It also seems as though this is an El Capitan problem.

Also, this technically might be a duplicate. But the answer the other post came up with was installing your own Python rather than relying on the default osx Python, which strikes me as more laborious.

---

Longer, Real Fix:

Eventually I ran into a situation where I actually needed to upgrade six in order to install some other libraries, at which point ignoring wasn't enough. The easiest way to upgrade six for me was to not use pip, but instead to manually download the .tar.gz from the six page, decompress, move to the newly decompressed six package directory, and run the installation manually (sudo, if necessary):

python setup.py install

Thanks to this answer for the guidance.

Selfconscious

I faced the same issue while installing tensorflow on El Capitan. Ignoring the previous installation worked for me as well. – Lynsey

sudo pip install awscli --ignore-installed six works for me. – Psyche

Answer 2

I don't think this is a duplicate, but actually this issue discussed here on the pip GitHub repository issues list.

NOTE: Depending on which package you're installing, you may need to modify the command referenced in the solution. In my case below, I was trying to setup virtualenv and virtualwrapper. If you're failing on installing aws-cli or any other package, just insert that into the script

The core of the problem is tied to Apple's new SIP that they shipped with El Capitan. More specifically,

OS X 10.11's python retains its own copy of six which is unremoveable, because of modifications Apple has done to their python distribution. 1.4.1 is not the latest, 1.10.0 is. It also comes early on their python's import path, so it will typically override later versions you install.

I would suggest using a different python for now. Python.org's, or installed via Homebrew, or Anaconda Python.

There is an incredibly detailed discussion on the Ask Different Stack Exchange that covers how the problems with SIP have been identified, addressed, and evolved since the original release of El Capitan. Although I found it fascinating, you'll spend less time following the instructions below than it would take you to read it, so I'd reccomend checking it out AFTER you finish the following...

I ran into the exact same error when attempting to upgrade VirtualEnv & VirtualEnvWrapper. There were several suggestions kicked around on that above thread, but in the end the most stable was to

1. Leverage the built-in support for the sudo OPTION to specify a HOME environment variable

$ man sudo

  -H   The -H (HOME) option option sets the HOME environment variable 
        to the home directory of the target user (root by default) as specified 
        HOME environment variable depends on sudoers(5) settings.  By default, 
        sudo will set HOME if env_reset or always_set_home are set, or if
        set_home is set and the -s option is specified on the command line.

2. Leverage pip's options to force an upgrade and ignore any pre-existing packages

$ pip install --help | grep upgrade

  -U, --upgrade   Upgrade all specified packages to the newest available 
        version. This process is recursive regardless of whether a dependency 
        is already satisfied.

beejhuff@ignatius:~/mac_setup$ pip install --help | grep ignore-installed

  -I, --ignore-installed  Ignore the installed packages (reinstalling instead).

First, my original attempt & error:

$ sudo pip install virtualenv virtualenvwrapper

The directory '/Users/beejhuff/Library/Caches/pip/http' or its parent directory 
is not owned by the current user and the cache has been disabled. 
Please check the permissions and owner of that directory. If executing 
pip with sudo, you may want sudo's -H flag.

The directory '/Users/beejhuff/Library/Caches/pip' or its parent directory 
is not owned by the current user and caching wheels has been disabled. 
check the permissions and owner of that directory. If executing pip with 
sudo, you may want sudo's -H flag.

Collecting virtualenv
  Downloading virtualenv-15.0.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 335kB/s 
Collecting virtualenvwrapper
  Downloading virtualenvwrapper-4.7.1-py2.py3-none-any.whl
Collecting virtualenv-clone (from virtualenvwrapper)
  Downloading virtualenv-clone-0.2.6.tar.gz
Collecting stevedore (from virtualenvwrapper)
  Downloading stevedore-1.12.0-py2.py3-none-any.whl
Collecting pbr>=1.6 (from stevedore->virtualenvwrapper)
  Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)
    100% |████████████████████████████████| 92kB 362kB/s 
Collecting six>=1.9.0 (from stevedore->virtualenvwrapper)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper
  Running setup.py install for virtualenv-clone ... done
  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_set.py", line 726, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_install.py", line 746, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
    copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-GQL8Gi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

The Solution

It required modifying my installation command in THREE specific ways:

1. I had to add the -H flag to sudo
2. I had to add the --upgrade option AFTER the name of the package I was upgrading (virtualenv)
3. I had to use the --ignore-installed flag and specify the six package was the one to be ignored.
4. *Note: general command is $ sudo -H pip install <packagename> --upgrade --ignore-installed six - replace<packagename> with the specific package you need to install`

Final Working Example

1st Upgrade virtualenv

$ sudo -H pip install virtualenv --upgrade --ignore-installed six

Password:

Collecting virtualenv
  Using cached virtualenv-15.0.0-py2.py3-none-any.whl
Collecting six
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: virtualenv, six
Successfully installed six-1.4.1 virtualenv-15.0.0

2nd Upgrade virtualenvwrapper

$ sudo -H pip install virtualenvwrapper --upgrade --ignore-installed six

Password:

  Downloading virtualenvwrapper-4.7.1-py2.py3-none-any.whl
Collecting six
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting virtualenv (from virtualenvwrapper)
  Downloading virtualenv-15.0.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 751kB/s 
Collecting virtualenv-clone (from virtualenvwrapper)
  Downloading virtualenv-clone-0.2.6.tar.gz
Collecting stevedore (from virtualenvwrapper)
  Downloading stevedore-1.12.0-py2.py3-none-any.whl
Collecting pbr>=1.6 (from stevedore->virtualenvwrapper)
  Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)
    100% |████████████████████████████████| 92kB 417kB/s 
Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper
  Running setup.py install for virtualenv-clone ... done
Successfully installed pbr-1.8.1 six-1.4.1 stevedore-1.12.0 virtualenv-15.0.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.1

Dallman

Worked for me! I was trying to install Tensorflow and it was hanging on an existing install of 'setuptools' although '--ignore-installed six' worked perfectly – Bessette

@Oliver Glad to hear it helped you out, Oliver & thanks for the up vote! – Antilebanon

This answer deserves more attention. It's a careful workaround for the problem without crude measures like resorting to sudoeing every package installation or setting up a second Python. – Crux

@eugene-a Thank you so much for that one recommendation, I sincerely appreciate it! – Antilebanon

Answer 3

Both of previous answers don't work for me. Finally, I got the solution from GitHub, aws/aws-cli, as excerpted below.

On OS X, if you see an error regarding the version of six that came with distutils in El Capitan, use the --ignore-installed option:

$ sudo pip install awscli --ignore-installed six

Psyche

I updated my answer to reflect that you need to specify the package you're trying to install. In original poster's example, it was scrape, in mine it was virtualenv and virtualwrapper, in yours it wads awscli. That was unclear earlier, would you say it makes more sense as a generic broadly applicable answer now? – Antilebanon

Answer 4

This would work!!

sudo pip install awsebcli --upgrade --ignore-installed six

Wilhelm

You should explain why they are getting the error in the first place, you should explain how you're command works and how it will fix the persons problem. So it may help the user understand and also extend to other users who might have the same problem in the future. – Regalia