Creating array of arrays in numpy with different dimensions

Question

I'm trying to create an array of numpy arrays, each one with a different dimension. So far, it seems to be fine. For example, if I run:

np.array([np.zeros((10,3)), np.zeros((11,8))])

the result is:

array([ array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]]),
       array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]])], dtype=object)

The dimension of the two matrices are completely different and the array is generated without any problem. However, if the first dimension of the two matrices is the same, it doesn't work anymore:

np.array([np.zeros((10,3)), np.zeros((10,8))])
Traceback (most recent call last):

  File "<ipython-input-123-97301e1424ae>", line 1, in <module>
    a=np.array([np.zeros((10,3)), np.zeros((10,8))])

ValueError: could not broadcast input array from shape (10,3) into shape (10)

What is going on?

Thank you!

Piggish

I have no good explanation other than it's a bug, but here is a workaround: L = [np.zeros((10,3)), np.zeros((10,8))] result = np.frompyfunc(L.__getitem__, 1, 1)(range(len(L))). – Kunlun

Thank you for the answer! It worked perfectly!! – Piggish

Answer 1

This has been hashed out before (Why do I get error trying to cast np.array(some_list) ValueError: could not broadcast input array; Numpy array in array with unequal length). Basically np.array does one of 3 things:

• make an n-dimensional array of a basic dtype, e.g. float.

• make an object dtype array

• raise an error, saying the first two are not possible.

These second two alternatives are fallback options, taken only if the first is impossible.

Without digging into the details of how the compiled code works, apparently what happens with

np.array([np.zeros((10,3)), np.zeros((10,8))])

is that it first sees the common first dimension, and deduces from that that it can take the first choice. It looks like it initialed a (10,2) array (2 items in your list), and tried to put the first array into the first row, hence the failed attempt to put a (10,3) array into a (10,) slot.

So if you really want an object dtype array, and not hit either the 1st or 3rd cases, you need to do some sort of 'round-about' creation.

PaulP and I have been exploring alternatives in Force numpy to create array of objects

Earlier: How to create a numpy array of lists?

In this question I suggest this iteration:

A=np.empty((3,),dtype=object)
for i,v in enumerate(A): A[i]=[v,i]

or in your case

In [451]: res = np.empty(2, object)
In [452]: alist = [np.zeros((10,3)), np.zeros((10,8))]
In [453]: for i,v in enumerate(alist):
     ...:     res[i] = v

Prevent numpy from creating a multidimensional array

---

Rather than iterate on alist, it may work to do:

res[:] = alist

It seems to work in most cases that I've tried, but don't be surprised if you broadcasting errors.

Mulderig