Cython: "fatal error: numpy/arrayobject.h: No such file or directory"

Question

I'm trying to speed up the answer here using Cython. I try to compile the code (after doing the cygwinccompiler.py hack explained here), but get a fatal error: numpy/arrayobject.h: No such file or directory...compilation terminated error. Can anyone tell me if it's a problem with my code, or some esoteric subtlety with Cython?

Below is my code.

import numpy as np
import scipy as sp
cimport numpy as np
cimport cython

cdef inline np.ndarray[np.int, ndim=1] fbincount(np.ndarray[np.int_t, ndim=1] x):
    cdef int m = np.amax(x)+1
    cdef int n = x.size
    cdef unsigned int i
    cdef np.ndarray[np.int_t, ndim=1] c = np.zeros(m, dtype=np.int)

    for i in xrange(n):
        c[<unsigned int>x[i]] += 1

    return c

cdef packed struct Point:
    np.float64_t f0, f1

@cython.boundscheck(False)
def sparsemaker(np.ndarray[np.float_t, ndim=2] X not None,
                np.ndarray[np.float_t, ndim=2] Y not None,
                np.ndarray[np.float_t, ndim=2] Z not None):

    cdef np.ndarray[np.float64_t, ndim=1] counts, factor
    cdef np.ndarray[np.int_t, ndim=1] row, col, repeats
    cdef np.ndarray[Point] indices

    cdef int x_, y_

    _, row = np.unique(X, return_inverse=True); x_ = _.size
    _, col = np.unique(Y, return_inverse=True); y_ = _.size
    indices = np.rec.fromarrays([row,col])
    _, repeats = np.unique(indices, return_inverse=True)
    counts = 1. / fbincount(repeats)
    Z.flat *= counts.take(repeats)

    return sp.sparse.csr_matrix((Z.flat,(row,col)), shape=(x_, y_)).toarray()

Squalene

can you add a tag for what OS you are using? – Tindall

@tcaswell 64-bit Windows 7. – Squalene

added the windows tag, hopefully that will help this problem be seen by people who know how to use windows (unlike me). – Tindall

I found this. Some of the terminology is above my head, but i'm gonna check it out. – Squalene

Does this answer your question? Make distutils look for numpy header files in the correct place – Acetate

Answer 1

In your setup.py, the Extension should have the argument include_dirs=[numpy.get_include()].

Also, you are missing np.import_array() in your code.

--

Example setup.py:

from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy

setup(
    ext_modules=[
        Extension("my_module", ["my_module.c"],
                  include_dirs=[numpy.get_include()]),
    ],
)

# Or, if you use cythonize() to make the ext_modules list,
# include_dirs can be passed to setup()

setup(
    ext_modules=cythonize("my_module.pyx"),
    include_dirs=[numpy.get_include()]
)    

Unapt

Why would i need np.import_array()? Isn't that for the Numpy C-API? – Squalene

I tried your idea, but now i get this crazy error that is too long to post here, but it starts with warning: untitled.pyx:8:49: Buffer unpacking not optimized away. – Squalene

Ah, true, you probably don't need np.import_array(). I rarely write numpy-using Cython extensions without it, though, so I use it as a matter of habit. As for your other problem, what you quoted is just a warning, not an error. If you have other errors that need fixing, please make a new post. – Unapt

include_dirs=[numpy.get_include()] is a nice trick thank you! – Brooking

definitely helped me building a point cloud library (PCL) python wrapper that is using Numpy and Cython, however, there are warnings Using deprecated NumPy API - is this deprecated? – Eckhart

What I advised is not deprecated, and it's possible that you aren't using any deprecated API functions either. The documentation describes why you get this message and what you can do about it. – Unapt

I've added a working example, as it wasn't clear to me where exactly I had to add the include_dirs directive. – Maurya

include_dirs passed to setup() gets ignored in the latest distutils, it has to be passed to each Extension, at least on mac – Pedicab

I get a AttributeError: 'module' object has no attribute 'import_array' error if doing np.import_array() in the python code calling the cython module... – Venator

Thanks for this; it seems this critical step is missing from the cython documentation! – Bethelbethena

Answer 2

For a one-file project like yours, another alternative is to use pyximport. You don't need to create a setup.py ... you don't need to even open a command line if you use IPython ... it's all very convenient. In your case, try running these commands in IPython or in a normal Python script:

import numpy
import pyximport
pyximport.install(setup_args={"script_args":["--compiler=mingw32"],
                              "include_dirs":numpy.get_include()},
                  reload_support=True)

import my_pyx_module

print my_pyx_module.some_function(...)
...

You may need to edit the compiler of course. This makes import and reload work the same for .pyx files as they work for .py files.

Source: http://wiki.cython.org/InstallingOnWindows

Wiedmann

Thanks, @SteveB. But can you elaborate a bit on what you mean by "For a one-file project like yours..."? The module above is one (albeit important) part of a larger application. How does pyximport affect my code's speed? And finally, the section here: "Since Cython 0.11, the pyximport module also has experimental compilation support for normal Python modules..." implies that it still has some kinks to work out. Can you explain that as well? – Squalene

Re "experimental compilation support for normal Python modules" -- with the code I suggested above, .py modules are compiled normally (not with cython) while .pyx modules are compiled with cython. If you pass pyimport = True into pyximport.install(), then it will use cython for everything, even for example import random or import os. I don't suggest using that feature, simply because there's no compelling reason to use it, and it could create problems. It's probably used mainly by cython developers. – Wiedmann

If pyximport works at all, it will create the exact same C code as any other method. So try it and see. I was referring to the fact that when the compilation process is sufficiently complicated, e.g. links to external system libraries, you might find that pyximport fails and you need a setup.py and cythonize to specify exactly how to build it. But the fact that your .pyx module has imports or cimports does not mean that it can't be compiled with pyximport; it may well be totally fine. – Wiedmann

This works great! But I recommend --compiler=msvc if you already have msvc installed. Then you don't need to install mingw. – Erigena

Answer 3

The error means that a numpy header file isn't being found during compilation.

Try doing export CFLAGS=-I/usr/lib/python2.7/site-packages/numpy/core/include/, and then compiling. This is a problem with a few different packages. There's a bug filed in ArchLinux for the same issue: https://bugs.archlinux.org/task/22326

Gaulish

Where do i add the export line? In my setup.py file? – Squalene

No, it's a shell command. Run it in your shell, then start compiling. – Gaulish

@NoobSaibot in the shell (where you run python setup.py) run the export .. command first. It sets the environmental variables of the shell, not anything directly to do with [pc]ython. – Tindall

@tcaswell: I figured as much. I'm using cmd, and got this 'export' is not recognized as an internal or external command, operable program or batch file. error...just can't win with this one... – Squalene

@NoobSaibot you are getting lunix answers for what smells like a windows problem.... – Tindall

Right... I'm sure there's a windows equivalent, I just have no idea what it is. – Gaulish

Actually, in Windows you should be able to add that path to the PATH environmental variable. Should be in system properties > environmental variables. – Gaulish

The problem is not specific to Windows. I just came across this same error on a Unix machine. – Eckhart

alternatively you can add this to the gcc command: /usr/bin/gcc -shared -fPIC -Os -I/usr/include/python3.5m/ -I/usr/lib/python3.5/site-packages/numpy/core/include/ -o mymodule.so mymodule.c – Finger

I have a Cython post you may be able to provide insight on. – Cancellate

This works for me in Windows, should be cross-platform. Add to your setup.py: import os; import numpy; os.putenv('INCLUDE', numpy.get_headers()) – Idzik

Answer 4

As per this answer, if you have installed numpy with pip on Linux, you will need to manually set a symbolic link to /usr/include/numpy

In my case the path is:

sudo ln -s /usr/local/lib/python3.8/dist-packages/numpy/core/include/numpy/ /usr/include/numpy

Welborn

Answer 5

If you are too lazy to write setup files and figure out the path for include directories, try cyper. It can compile your Cython code and set include_dirs for Numpy automatically.

Load your code into a string, then simply run cymodule = cyper.inline(code_string), then your function is available as cymodule.sparsemaker instantaneously. Something like this

code = open(your_pyx_file).read()
cymodule = cyper.inline(code)

cymodule.sparsemaker(...)
# do what you want with your function

You can install cyper via pip install cyper.

Norwegian

Answer 6

Simple answer

A way simpler way is to add the path to your file distutils.cfg. It's path behalf of Windows 7 is by default C:\Python27\Lib\distutils\. You just assert the following contents and it should work out:

[build_ext]
include_dirs= C:\Python27\Lib\site-packages\numpy\core\include

Entire config file

To give you an example how the config file could look like, my entire file reads:

[build]
compiler = mingw32

[build_ext]
include_dirs= C:\Python27\Lib\site-packages\numpy\core\include
compiler = mingw32

Srini

Answer 7

It should be able to do it within cythonize() function as mentioned here, but it doesn't work beacuse there is a known issue

Dealing

Answer 8

I hadn't sudo privileges on the server I was running and export CFLAGS didn't work with me. For sake of simplicity, I've installed Anaconda ( https://docs.anaconda.com/anaconda/install/) which creates links to all its installed packages, including Numpy. You can also install miniconda and work with environments to avoid using too much space.

Jeromyjerreed

Are you sure this answers the question? Please read How to Answer. – Eckhart