numexpr.evaluate("a+b",out=a)

Question

Is it safe in python numexpr to assign values to the same array you are operating on to avoid creating a temporary array?

From the description of memory usage on the project homepage it looks okay, but without diving into the source code, that is hardly a solid answer.

I tried the following which works fine, but I am hoping for confirmation from someone more familiar with this package:

import numpy as np
import numexpr as ne
a = np.ones(5)
b = a.copy()
ne.evaluate("a+b",out=a)
array([ 2.,  2.,  2.,  2.,  2.])

Caruso

I can't think of a case where it wouldn't be safe off the top of my head (a lot of indexing tricks that would potentially cause problems don't work in numexpr to begin with). On a side note, for simple cases like your example it's also useful to be aware of: np.add(a, b, out=a). numexpr is extremely nice, but it is possible to avoid temporary arrays without it, if you don't want another dependency. – Engrail

Answer 1

It works, because numexpr still uses temporary arrays internally, albeit in chunk sizes of 1024 elements (or 4096 if using VML). You can think of these chunks of the inputs as slices, though they are stored as appropriate C data types for speed and memory compactness during the evaluation. The results will be stored into the out parameter after the calculation of each chunk is performed, otherwise it must allocate an array of of the same size as the inputs.

Check out the section Why It Works for pseudocode of how numexpr evaluates vectorized arithmetic.

Disruption