Why can't I get `pip install lxml` to work within a virtualenv?

Question

Note: I'm using virtualenvwrapper.

Before activating the virtual environment:

$ pip install lxml
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/lib/python2.7/dist-packages  
Cleaning up...  

After activating the virtual environment:

(test-env)$ pip install lxml
force/build/lxml/src/lxml/includes/etree_defs.h:9:31: fatal error: 
libxml/xmlversion.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

----------------------------------------
Command /home/chaz/dev/envs/test-with-system-python-force/bin/python2
.7 -c "import setuptools;__file__='/home/chaz/dev/envs/test-with-
system-python-force/build/lxml/setup.py';exec(compile(open(__file__).
read().replace('\r\n', '\n'), __file__, 'exec'))" install --record 
/tmp/pip-bJ6Q_B-record/install-record.txt --single-version-externally
-managed --install-headers /home/chaz/dev/envs/test-env/include/site/python2.7 failed with error code 1 in 
/home/chaz/dev/envs/test-env/build/lxml
Storing complete log in /home/chaz/.pip/pip.log

Colet

I somehow fail to see in which way this is a specific programming question. IMHO, this belongs to serverfault.com – Suspend

@MarkusWMahlberg Virtualenv is an extremely common Python utility which is primarily used by programmers during development, not by sysadmins during deployment. lxml is very popular, and this type of issue is not rare. I consider this to be on topic under the criteria of "software tools commonly used by programmers". – Kym

@JeremyBanks I am well aware of virtualenv. But I think we have a rather environment related problem and chances for getting a usefull answer should be much higer on serverfault. – Suspend

Answer 1

You probably already have lxml installed on your system, perhaps installed due to a system package. Thus, the first attempt (pip install lxml without an active virtualenv) doesn't fail, but it also doesn't install it; it really doesn't do anything.

In a virtualenv, by default, the system packages are ignored. Therefore, pip thinks that lxml is not installed. Therefore, it tries to install it into your virtual environment.

lxml contains C modules that need to be compiled in order to install properly. However, the compilation of those C modules rely on your having some "development libraries" already installed as well. These development libraries are C libraries, not Python, and as such pip won't be able to automatically fetch them from the internet and install them for you.

Therefore, you will need to install these development libraries on your own, most likely using your package manager. In a Debian system (like Ubuntu), this is...

apt-get install libxml2-dev libxslt-dev

This will install the libxml2 and libxslt development libraries to your local system. If you try again to install lxml, the C module compilation step should work because now these development libraries are on your system.

The error message you were receiving was due to the fact that these libraries were missing (the libxml/xmlversion.h: No such file or directory part of the error message).

See also: How to install lxml on Ubuntu

Shope

Successfully installed lxml Thank you! I'm curious if you know of a place that maps the necessary headers to various (popular) Python packages. I've had problems with other also. Also, does the apt-get python-dev only get headers for the standard library? – Colet

Best bet: search google for the header file that is missing, hope someone else is having the same problem. Also, yes, python-dev only gets you headers for the standard library. libxml2-dev has nothing to do with python: libxml2 is a C library that you just happen to be using a python library to work with. That is why it wasn't installed when you installed python-dev. Otherwise, you would basically need to install every C library in existence in the world to make sure you have all the headers that you would need to make every python library (such as lxml) works without additional steps – Shope

hi! What if I can't install the devs? I'm not an administrator of the machine and I badly need this now.. thanks – Surbased

@ozama I would recommend you describe your situation in a new question for others to answer. – Shope

Answer 2

for centos users: when getting:

error: command 'gcc' failed with exit status 1

DO:

sudo yum install libxslt-devel libxml2-devel

Anatomical

Answer 3

If you have lxml installed at the system level, and want to migrate it into a virtualenv that you didn't create with --system-site-packages, you can symlink it into your virtualenv's dist-packages folder.

Outside your virtualenv, in a python shell:

import lxml
print lxml.__file__

In my case, it's found in /usr/lib/python2.7/dist-packages. There'll be an lxml folder and egg-info file. Wherever your virtualenv is, go into its /lib/python-x.y/dist-packages folder (you may need to create dist-packages), and symlink both the library folder and egg into it.

Flagship

Answer 4

You're most likely looking for this: Microsoft Visual C++ 14.0 is required (Unable to find vcvarsall.bat)

Look for the visual studio website and go into: "Tools for visual studio" at the bottom, expand it by clicking. Select Download next to "Build Tools for Visual Studio 2017" at the top.

Saxecoburggotha

Um, OP is clearly using some version of Unix or Linux, not Microsoft Windows … – Careycarfare