Python interpteter can't find module in virtualenv, but pip sees it and won't install

Question

I'm trying to use the pytools module within the virtualenv created by Nervana for their Neon deep learning package, but can't seem to either find pytools or pip it. When I enter my virtualenv, I see this behavior:

me@ARL--M6800:~/Downloads/neon$ source .venv/bin/activate
(.venv) me@ARL--M6800:~/Downloads/neon$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import pytools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ImportError: No module named pytools
>>> 

>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/dist-packages', '/home/me/Downloads/neon',
 '/home/me/Downloads/neon/.venv/lib/python2.7',
 '/home/me/Downloads/neon/.venv/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/me/Downloads/neon/.venv/lib/python2.7/lib-tk',
 '/home/me/Downloads/neon/.venv/lib/python2.7/lib-old', 
 '/home/me/Downloads/neon/.venv/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7', 
 '/usr/lib/python2.7/plat-x86_64-linux-gnu', 
 '/usr/lib/python2.7/lib-tk', 
 '/home/me/Downloads/neon/.venv/local/lib/python2.7/site-packages', 
 '/home/me/Downloads/neon/.venv/lib/python2.7/site-packages']



(.venv) me@ARL--M6800:~/Downloads/neon$ pip install pytools 
Requirement already satisfied (use --upgrade to upgrade): pytools in     
 /usr/local/lib/python2.7/dist-packages/pytools-2016.1-py2.7.egg
Requirement already satisfied (use --upgrade to upgrade): 
 decorator>=3.2.0 in /usr/local/lib/python2.7/dist-packages (from pytools)
Requirement already satisfied (use --upgrade to upgrade): appdirs>=1.4.0 
  in /usr/local/lib/python2.7/dist-packages/appdirs-1.4.0-py2.7.egg (from 
  pytools)
Requirement already satisfied (use --upgrade to upgrade): six>=1.8.0 in 
 /usr/local/lib/python2.7/dist-packages (from pytools)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6.0 in 
 /usr/local/lib/python2.7/dist-packages (from pytools)

So, I can't import pytools becauseit isn't on my sys.path. According to pip, it is installed in the /usr/local/lib/python2.7/dist-packages/pytools-2016.1-py2.7.egg directory, which leaves me with 3 questions:

First: Why can my virtualenv see my system-wide packages? I thought the default was not to see them. When I look at the Makefile used to create the virtualenv, I see this

  # where our installed python packages will live
  VIRTUALENV_DIR := .venv
  VIRTUALENV_EXE := virtualenv -p python2.7  # use pyvenv for python3 install
  ACTIVATE := $(VIRTUALENV_DIR)/bin/activate

which should give me default behavior.

Second: Why are there egg directories in my dist-packages dir? Doesn't this make it harder to find those modules? (Though apparently, the sys.path for my system environment python has been updated to search in the egg dir. When/How???)

Third: What is an efficient way of fixing things so that my virtualenv will have access to pytools?

(I would've numbered my list instead of First/Second/Third, but then the Makefile code I inserted didn't format well)

Breathed

You might want to try running "which pip` to see whether you are running the global pip command or the venv specific pip command. You should be getting the venv specific copy (at $(VIRTUALENV_DIR)/bin/pip). If not, then that is your problem. – Bases

@Bases - Thanks. That's not it, but it's good to know of one other thing tocheck. – Breathed

Answer 1

For what it's worth, I uninstalled pytools using pip in my main environment. Then, went into my virtual environment and used pip to install pytools. Finally, went back to my main environment and reinstalled pytools using pip install.

Now things are working. (Though oddly, pytools is no longer installed in a '.egg' directory. I think this is better, but am now even more puzzled as to why/when things get installed into .egg directories)

Breathed

I can confirm that. If a module has been installed as egg, it is not recognized by the python interpreter in the virtualenv. – Eskew

commenting here to confirm that reinstalling a new venv with all the requisites and not bringing the cached version from root fixed my problem with another module (pymysql) – Ping

Answer 2

First of all, I don't know anything about Nervana's Neon.

I get the same sys.path when the virtualenv is created with root privileges:

dm@Z580:~$ sudo virtualenv test1 -p python2.7
[sudo] password for dm: 
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in test1/bin/python2.7
Also creating executable in test1/bin/python
Installing setuptools, pip, wheel...done.
dm@Z580:~$ source test1/bin/activate
(test1)dm@Z580:~$ which python
/home/dm/test1/bin/python
(test1)dm@Z580:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/home/dm/test1/lib/python2.7',
 '/home/dm/test1/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/dm/test1/lib/python2.7/lib-tk',
 '/home/dm/test1/lib/python2.7/lib-old',
 '/home/dm/test1/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/home/dm/test1/local/lib/python2.7/site-packages',
 '/home/dm/test1/lib/python2.7/site-packages']

I would remove the current virtualenv, recreate it without sudo and then install all requirements with the virtualenv's pip.

cd ~/Downloads
rm -rf neon/.venv/
virtualenv neon/.venv/ -p python2.7
source neon/.venv/bin/activate
pip install -r neon/requirements.txt

Smoothspoken