Numexpr doesn't recognize float type (sparse matrix)

Question

I would like to evaluate the performance of numexpr module in python (2.7). For that purpose, I created a random sparse matrix of size (10^5, 10^5). However, the script below throws an error at the expression evaluation step already, saying that it doesn't recognize the object type.

What am I doing wrong?

Code:

import timeit
import scipy.sparse as sps
import numpy as np
import numexpr as ne

test_matrix = sps.rand(1e4, 1e4, density=0.01, format='coo', dtype = np.float32)
ne.evaluate('sum(test_matrix, axis = 1)')

setup = 'import numexpr as ne; import numpy as np'
print min(timeit.Timer('ne.evaluate(sum(test_matrix, axis = 1))', setup=setup).repeat(7, 1000))

Error:

Traceback (most recent call last):

File "benchmark_expressmath.py", line 19, in <module>
ne.evaluate('sum(test_matrix, axis = 1)')
File "C:\Users\blahblah\AppData\Local\Continuum\Anaconda\lib\site-packages\numexpr\necompiler.py", line 756, in evaluate
signature = [(name, getType(arg)) for (name, arg) in zip(names, arguments)]
File "C:\Users\blahblah\AppData\Local\Continuum\Anaconda\lib\site-packages\numexpr\necompiler.py", line 654, in getType
raise ValueError("unknown type %s" % a.dtype.name)
ValueError: unknown type object

Frangipane

Answer 1

numexpr expects the variables to be numpy arrays. It doesn't handle scipy's sparse matrices. (See, for example, this email thread: http://numpy-discussion.10968.n7.nabble.com/ANN-numexpr-2-3-final-released-td36154.html)

Garrott

Do you know of any similar packages for Python that can handle sparse matrices? – Frangipane

There's pysparse, but I haven't tried it, and I don't know how its performance compares to scipy. – Garrott

pysparse doesn't AFAIK have numexpr-like capabilities. – Choir

@pv., no, I wouldn't think so, but if the OP's reason for investigating numexpr is to improve performance, and if it turned out that pysparse was much faster than scipy, the need for numexpr would be reduced. – Garrott