Compiler problems with pip during numpy install under Windows 8.1, 7 Enterprise and 7 Home Editions

Question

I am unable to install numpy via pip install numpy on my computer running Python 3.4 due to various errors I receive linked to compilation issues (This is only the case on a 64-bit installation of Python).

This is a problem that has been reported extensively and I had a related question about unable to find vcvarsall.bat. I have tested this on three different clean machines running Windows 8.1, 7 Enterprise and 7 Home Editions and it always comes up.

Installing Visual Studio 2010 Express C++ gets rid of the first error in the link - i.e. Unable to find vcvarsall.bat but throws out a next exception ending with a ValueError as here:

  File "C:\Python34\lib\distutils\msvc9compiler.py", line 371, in initialize

    vc_env = query_vcvarsall(VERSION, plat_spec)

  File "C:\Python34\lib\distutils\msvc9compiler.py", line 287, in query_vcvarsall

    raise ValueError(str(list(result.keys())))

ValueError: ['path']

I have then followed this advice and patched the file as linked in the discussion forum which resulted in a KEY_BASE error.

  File "C:\Users\Matej\AppData\Local\Temp\pip_build_Matej\numpy\numpy\distutils\command\config.py", line 18, in <module>

    from numpy.distutils.mingw32ccompiler import generate_manifest

  File "C:\Users\Matej\AppData\Local\Temp\pip_build_Matej\numpy\numpy\distutils\mingw32ccompiler.py", line 36, in <module>

    from distutils.msvccompiler import get_build_version as get_build_msvc_version

  File "C:\Python34\lib\distutils\msvccompiler.py", line 638, in <module>

    from distutils.msvc9compiler import MSVCCompiler

  File "C:\Python34\lib\distutils\msvc9compiler.py", line 71, in <module>

    r"v%sA"

  File "C:\Python34\lib\distutils\msvc9compiler.py", line 67, in <listcomp>

    WINSDK_PATH_KEYS = [KEY_BASE + "Microsoft SDKs\\Windows\\" + rest for rest in (

NameError: name 'KEY_BASE' is not defined

Following the advice in the same link, I have added the following definition of KEY_BASE before the variable gets called in msvc9compiler.py:

KEY_BASE = r"Software\Microsoft\\"

Which results in the final error I was not able to troubleshoot:

  File "C:\Users\Matej\AppData\Local\Temp\pip_build_Matej\numpy\numpy\distutils\command\build_src.py", line 164, in build_sources

    self.build_library_sources(*libname_info)

  File "C:\Users\Matej\AppData\Local\Temp\pip_build_Matej\numpy\numpy\distutils\command\build_src.py", line 299, in build_library_sources

    sources = self.generate_sources(sources, (lib_name, build_info))

  File "C:\Users\Matej\AppData\Local\Temp\pip_build_Matej\numpy\numpy\distutils\command\build_src.py", line 386, in generate_sources

    source = func(extension, build_dir)

  File "numpy\core\setup.py", line 682, in get_mathlib_info

    raise RuntimeError("Broken toolchain: cannot link a simple C program")

RuntimeError: Broken toolchain: cannot link a simple C program

I have tried the following but none of it resolved the Broken toolchain error:

• This link that includes a further patch to msvc9compiler.py
• This link by Peter Cock (This helps on the 32-bit install of Python3.4)
• Tried installing using easy_install as some users suggested it might work that way

NOTE 1: I am aware of the workaround using the compiled binaries on this website I am specifically looking for solution using distutils if there is one. NOTE 2: The error logs are obviously larger and I cut them for readability.

Demodulation

don't forget to run cmd as administrator – Nathanaelnathanial

Answer 1

I was able to reproduce all these errors in Windows 7 Professional (64 bit).

Your final issue (Broken toolchain) is caused by more manifest related issues. I was able to work around this by changing the following line (in msvc9compiler.py):

mfinfo = self.manifest_get_embed_info(target_desc, ld_args)

to

mfinfo = None

thus bypassing the if statement which immediately follows. After this change numpy successfully compiled for me.

Notional

I have tried this already as it was suggested in Peter Cock's link above but it didn't work for me. Did you follow my procedure? I.e. patching msvc9compiler.py and then inserting KEY_BASE followed by change in mfinfo? I am thinking that maybe I have made a mistake in one of the steps. – Demodulation

I made the changes described in msvccompiler9_33.diff by steve.dower in the link you gave. Then I added the KEY_BASE just before all the HKEYS are set (near the beginning of the file). Then I changed the mfinfo line as I said (sorry I did not see the link to Peter Cock's post), and it compiled. – Notional

I am not sure if this is relevant but are you using Visual Studio 2010 Express? – Demodulation

No, I use Visual Studio 2010 Professional. It's possible there's an issue with 2010 Express on 64-bit systems. – Notional

Thanks, I will check that and see if that resolves the issue. – Demodulation

I can confirm that it is the manifest issue on the 32-bit installation of Python but not the 64-bit version. I am unsure why. I have upvoted your answer and edited the question :). – Demodulation

Answer 2

Although the accepted answer is working, it's an unnecessary monkey-patch to a battle-tested library.

To fix the issue at it's core: Simply add (x86 or x64 - depending on your target) Microsoft Manifest Tool (mt.exe) to your path and install numpy through pip.

Since Visual Studio 2017, mt.exe has moved to the Windows SDK 7.1/8.1/10, which needs to be installed.

My mt.exe (x86) is located in: C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x86 so that's what I have added to the path, do that ether through Control Panel > System > Advanced system settings > Environment Variables or by using the pathman command in the console (cmd.exe).

Hope it helps, happy compiling!

Antonomasia

What do you mean, add it to the path? Just where python and pip are being run? – Hagerman

Jake Gaston, It's the PATH environment variable: How to: Add Tool Locations to the PATH Environment Variable – Antonomasia

Answer 3

If you didn't fixed the issue with those steps please check this simple solution that helped me to build numpy: Go to Jurko's repository * in which he is patching/improving the scripts for building with MSVC and download/copy paste the files:

msvc9compiler.py
msvccompiler.py

Links:

• https://bitbucket.org/jurko/cpython/src/b0962aec201e7d4b4f1ac32dbc130379ec6d943a/Lib/distutils/msvc9compiler.py?at=jurko/distutils_msvc_express_fix

• https://bitbucket.org/jurko/cpython/src/b0962aec201e7d4b4f1ac32dbc130379ec6d943a/Lib/distutils/msvccompiler.py?at=jurko/distutils_msvc_express_fix

Save them in the folder '\Lib\distutils\' of your python installation directory, eg: "C:\WinPython-64bit-3.4.3.2\python-3.4.3.amd64\Lib\distutils".

There are several other solutions related to this issue of building python modules with pip/easy_install, eg: error: Unable to find vcvarsall.bat ; Unable to find vcvarsall.bat using Python 3.3 in Windows 8 that seem unsolvable sometimes.

My system: windows 7, windows sdk 7.1, vs2010 express with sp1, python 3.4.3, 64 bit

Jacquiline

Answer 4

I think the easiest solution will be to open the Visual Studio command prompt (you should not need any other fixes, just the latest version of Python 2 or 3).

Specifically, in your Start menu, you should have something like "Visual Studio Command Prompt (2010)". Open this program (it will open a command prompt), then in the command prompt, type:

pip install numpy

Comments:

1) Different versions of Python are meant to be compiled with different versions of Visual Studio. I think that the Python guys intend for you to compile with the 2010 version (I could be wrong about this), so trying to open the Visual Studio 2013 prompt might not work. It could be that the newest version of Python switched compilers, but I would start by trying the 2010 command prompt.

2) You might have multiple command prompt options. I have had luck with just the bare bones version "Visual Studio Command Prompt (2010)"

3) To install Python 3.4, I could simply open a standard command prompt, and then

pip install numpy

but for Python 2.7, I had to use this command prompt trick.

Mylor

Used this approach for cpython 3.5.1 amd64, numpy-1.10.4 with the preview release of c++ 2015 and it worked. – Swink