Building lxml for Python 2.7 on Windows

Question

I am trying to build lxml for Python 2.7 on Windows 64 bit machine. I couldn't find lxml egg for Python 2.7 version. So I am compiling it from sources. I am following instructions on this site

http://lxml.de/build.html

under static linking section. I am getting error

C:\Documents and Settings\Administrator\Desktop\lxmlpackage\lxml-2.2.6\lxml-2.2.
6>python setup.py bdist_wininst --static
Building lxml version 2.2.6.
NOTE: Trying to build without Cython, pre-generated 'src/lxml/lxml.etree.c' need
s to be available.
ERROR: 'xslt-config' is not recognized as an internal or external command,
operable program or batch file.

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt
Building against libxml2/libxslt in one of the following directories:
  ..\libxml2-2.7.6--win32--w2k--x64\lib
  ..\libxslt-1.1.26--win32--w2k--x64--0002\lib
  ..\zlib-1.2.4--win32--w2k--x64
  ..\iconv-1.9.1--win32--w2k--x64-0001\lib
running bdist_wininst
running build
running build_py
running build_ext
building 'lxml.etree' extension
error: Unable to find vcvarsall.bat

Can any one help me with this? I tried setting the path to have Microsoft Visual Studio.. I can run vcvarsall.bat from the commandline.. but python is having problems

Anastase

If you happen to use AMD64 you could try these: lfd.uci.edu/~gohlke/pythonlibs – Century

@Christo Do you have instructions somewhere on how you generated these executables? – Anastase

I didn't compile these, so the best thing to do would be to contact the author: lfd.uci.edu/~gohlke – Century

What version of Visual Studio do you use? – Sitwell

Answer 1

I bet you're not using VS 2008 for this :)

There's def find_vcvarsall(version): function (guess what, it looks for vcvarsall.bat) in distutils with the following comment

At first it tries to find the productdir of VS 2008 in the registry. If that fails it falls back to the VS90COMNTOOLS env var.

If you're not using VS 2008 then you have neither the registry key nor suitable environment variable and that's why distutils can't find vcvarsall.bat file. It does not check if the bat file is reachable through the PATH environment variable.

The solution is to define VS90COMNTOOLS variable to point to Tools directory of Visual Studio.

That being said take a look at 11.4. distutils.msvccompiler — Microsoft Compiler section in Python's docs which states

Typically, extension modules need to be compiled with the same compiler that was used to compile Python.

Martin v. Loewis in the email titled Download Visual Studio Express 2008 now on python-list mailing list states the same

Python 2.6, 2.7, and 3.1 are all built with that release (i.e. 2008). Because of another long tradition, Python extension modules must be built with the same compiler version (more specifically, CRT version) as Python itself. So to build extension modules for any of these releases, you need to have a copy of VS 2008 or VS 2008 Express.

In the light of above statements you should use VS 2008 if you want to build lxml for Python 2.7 so although setting VS90COMNTOOLS takes care of finding vcvarsall.bat file it's not the solution.

That being said :) people do try to use older CRT with newer compiler:
Can I use Visual Studio 2010's C++ compiler with Visual Studio 2008's C++ Runtime Library?
How to Enforce C++ compiler to use specific CRT version?
VS 2008 - Link against older C runtime

I'd like to thank Kev Dwyer (for pointing out importance of version of VS which is used) and Stefan Behnel (for pointing me to distutils as a place dealing with compiler's configuration) in the thread Problem building lxml under Windows - error: Unable to find vcvarsall.bat on lxml mailing list. I'd like to also thank agronholm from freenode #distutils IRC channel for confirmation that distutils does contain code which looks for vcvarsall.bat file.

Sitwell

okay, you explained it, but so what needs to be done in the absence of vs2008? – Vigue

@Maysam You shouldn't use anything else but Visual Studio 2008 for this. – Sitwell

Ok long and winding answer... But it still doesn't explain how to actually fix the problem that the guy was having? – Ballyhoo

@ZoranPavlovic But it still doesn't explain how to actually fix the problem that the guy was having? Sure, it does. – Sitwell

I somehow traced the values of variables in distutils\msvc9compiler.py, now the toolskey in function find_vcvarsall(version) gives VS100COMNTOOLS. Does it mean that I should install Visual Studio 2010 Express instead of 2008? – Paleozoology

@ziyuang This would mean you use Python 3.3 which uses Microsoft Visual Studio 2010. If that's the case then the answer is yes, you should install this version. – Sitwell

@PiotrDobrogost Thanks. Then I meet with this ValueError: ['path'] error: https://mcmap.net/q/13232/-value-error-trying-to-install-python-for-windows-extensions/688080 . It seems I should install a professional version instead of an express one. – Paleozoology

So, if I manage to get VS2008, how do I set the path so that it finds it? [I have VS2008] – Ballata

Visual C++ 2008 Express is available here go.microsoft.com/?linkid=7729279 – Tommietommy

Visual C++ 2010 Express is available here go.microsoft.com/?linkid=9709949 – Tommietommy

Answer 2

After following the recommended solution:

1. downloading VCForPython27.msi from Microsoft,
2. installing it (Win7, Python(x, y) 2.7.9 32bit),
3. entering / updating the environment variable VS90COMNTOOLS to the installation directory value (C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0)

my problem still existed (want to build a Python extension in C).

I had to do the following 2 unbelievably dirty tweaks, before everything now is indeed working:

1. modify "msvc9compiler.py" in "C:\Python27\Lib\distutils", function find_vcvarsall, to now point to "Visual C++ for Python" instead of to "VC".
2. copy the directories founder under "C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0\" to "C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\" (i.e. one dir level up).

I cannot tell who was doing something wrong here - probably I.

EDIT. Moving directories works because of the issue described in this distutils bug.

even if VS90COMNTOOLS is set, msvc9compiler isn't able to find vcvarsall.bat because it is installed in %installdir%/vcvarsall.bat and not %installdir%/VC/vcvarsall.bat

The described workaround is using the Visual C++ command prompt:

1. Enter MSVC for Python command prompt

2. SET DISTUTILS_USE_SDK=1

3. SET MSSdk=1

4. python.exe setup.py ...

Champagne

Answer 3

Jorj McKie was almost correct: indeed installing VCForPython27.msi isn't enough, and yes there is an issue in distutils which prevent it from finding find_vcvarsall. In fact the issue is not directly in distutils, but in how VCForPython27.msi was packaged and where vcvarsall.bat is placed (the folders layout is different from the VS2008 SDK).

A simple workaround meanwhile this gets patched maybe in Python 2.7.11: use setuptools instead of distutils.

Another manual workaround if you're stuck with distutils:

1) Enter MSVC for Python command prompt
2) SET DISTUTILS_USE_SDK=1
3) SET MSSdk=1
4) you can then build your C extensions: python.exe setup.py ...

Bug report and workaround by Gregory Szorc: http://bugs.python.org/issue23246

More info and a workaround for using %%cython magic inside IPython: https://github.com/cython/cython/wiki/CythonExtensionsOnWindows

Deane

Answer 4

https://github.com/develersrl/gccwinbinaries

I had similar issues. This worked instantly without anything other than using an install wizard and setting one preference.

Blowup