generalized cumulative functions in NumPy/SciPy?

Question

Is there a function in numpy or scipy (or some other library) that generalizes the idea of cumsum and cumprod to arbitrary function. For example, consider the (theoretical) function

cumf( func, array) 

func is a function that accepts two floats, and returns a float. Particular cases

lambda x,y: x+y 

and

lambda x,y: x*y 

are cumsum and cumprod respectively. For example, if

func = lambda x,prev_x: x^2*prev_x 

and I apply it to:

cumf(func, np.array( 1, 2, 3) )

I would like

np.array( 1, 4, 9*4 )

Ringster

I think this is a great idea, but I think it is more canonical to define func = lambda x, y: x+y (two non-related arguments) and let the cumulative concept to "figure out" that x and y are actually consecutive elements of a sequence. – Os

that was more for human readers ;) – Ringster

Answer 1

NumPy's ufuncs have accumulate():

In [22]: np.multiply.accumulate([[1, 2, 3], [4, 5, 6]], axis=1)
Out[22]: 
array([[  1,   2,   6],
       [  4,  20, 120]])

Unfortunately, calling accumulate() on a frompyfunc()'ed Python function fails with a strange error:

In [32]: uadd = np.frompyfunc(lambda x, y: x + y, 2, 1)

In [33]: uadd.accumulate([1, 2, 3])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

ValueError: could not find a matching type for <lambda> (vectorized).accumulate, 
            requested type has type code 'l'

This is using NumPy 1.6.1 with Python 2.7.3.

Drummond

Ah, but I would like an numpy array as output – Ringster

@Ringster -- What do you mean you'd like a numpy array as output? A concrete example would be helpful. – Ouster

No need for np.array here. Numpy will happily operate on your lists, but good answer (+1) – Ouster

While it fails with lambda functoins, I can recreate my desired function with ufuncs. This is what I needed. Thanks for introducing me to ufuncs and accumulate! – Ringster

@Ringster Now that you figured out how to do it with ufuncs and accumulate, please post a piece of working code as your own answer, or editing the question, so that others will benefit also. Thanks! – Os

On further analysis, I actually cannot use this method. There is apparently a bug in Numpy 1.6, hence the ValueEror above (same as I get) mail.scipy.org/pipermail/numpy-discussion/2011-September/… Either way, my question was answered with the knowledge of frompyfunc() and accumulate() – Ringster

@Cam.Davidson.Pilon: Yes, that's exactly the bug I've run into. – Drummond

with numpy 1.24.2 the error is fixed – Atmosphere

Answer 2

The ValueError above is still a bug using Numpy 1.20.1 (with Python 3.9.1).

Luckily a workaround was discovered that uses casting: https://groups.google.com/forum/#!topic/numpy/JgUltPe2hqw

import numpy as np
uadd = np.frompyfunc(lambda x, y: x + y, 2, 1)
uadd.accumulate([1,2,3], dtype=object).astype(int)
# array([1, 3, 6])

Note that since the custom operation works on an object type, it won't benefit from the efficient memory management of numpy. So the operation may be slower than one that didn't need casting to object for extremely large arrays.

Tiler