How to create a numpy array of lists?

Question

I want to create a numpy array in which each element must be a list, so later I can append new elements to each.

I have looked on google and here on stack overflow already, yet it seems nowhere to be found.

Main issue is that numpy assumes your list must become an array, but that is not what I am looking for.

Chiachiack

Why not create a 2D-array? – Energid

why numpy array of lists? why not list of numpy arrays? or list of lists? – Bauhaus

I have special requirements. – Chiachiack

Please explain your "special requirements" in more detail. If your primary concern is with the speed of append operations then you can't do much better than a regular Python list-of-lists, since appending to a list is very cheap compared with array concatenation. However, this comes at a big storage and performance cost assuming that you want to perform numerical operations on your list-of-lists. Is every sub-list going to have the same length, or are you trying to represent a 'ragged' array with different row lengths? – Edify

There are lots of SO questions about creating arrays of dtype=object. – Counterattraction

@ali_m, adjacency list for sparse graphs. I need the operations to be O(m+n). lists of lists kill that. Also each list may have different sizes. I need the first vector to be an array of O(1) for accessing each element, and then a list to access it at O(d_max) for each element of the given list. – Chiachiack

Please add the full details to your question. I'm now even more convinced that you want a list-of-lists. Appending and indexing are both O(1) for Python lists. Concatenating numpy arrays is O(n). – Edify

In fact, here's an example from the BDFL himself, suggesting a graph implementation using Python lists to store adjacency information. – Edify

Appending and indexing you are right, searching is not. – Chiachiack

Also in terms of efficienty, numpy arrays are more memory efficient than python lists. This is why I wanted a numpy array and then use lists only for what it is really needed. – Chiachiack

Answer 1

As you discovered, np.array tries to create a 2d array when given something like

 A = np.array([[1,2],[3,4]],dtype=object)

You have apply some tricks to get around this default behavior.

One is to make the sublists variable in length. It can't make a 2d array from these, so it resorts to the object array:

In [43]: A=np.array([[1,2],[],[1,2,3,4]])
In [44]: A
Out[44]: array([[1, 2], [], [1, 2, 3, 4]], dtype=object)

And you can then append values to each of those lists:

In [45]: for i in A: i.append(34)
In [46]: A
Out[46]: array([[1, 2, 34], [34], [1, 2, 3, 4, 34]], dtype=object)

np.empty also creates an object array:

In [47]: A=np.empty((3,),dtype=object)
In [48]: A
Out[48]: array([None, None, None], dtype=object)

But you then have to be careful how you change the elements to lists. np.fill is tempting, but has problems:

In [49]: A.fill([])
In [50]: A
Out[50]: array([[], [], []], dtype=object)
In [51]: for i in A: i.append(34)
In [52]: A
Out[52]: array([[34, 34, 34], [34, 34, 34], [34, 34, 34]], dtype=object)

It turns out that fill puts the same list in all slots, so modifying one modifies all the others. You can get the same problem with a list of lists:

In [53]: B=[[]]*3
In [54]: B
Out[54]: [[], [], []]
In [55]: for i in B: i.append(34)
In [56]: B
Out[56]: [[34, 34, 34], [34, 34, 34], [34, 34, 34]]

The proper way to initial the empty A is with an iteration, e.g.

In [65]: A=np.empty((3,),dtype=object)
In [66]: for i,v in enumerate(A): A[i]=[v,i]
In [67]: A
Out[67]: array([[None, 0], [None, 1], [None, 2]], dtype=object)
In [68]: for v in A: v.append(34)
In [69]: A
Out[69]: array([[None, 0, 34], [None, 1, 34], [None, 2, 34]], dtype=object)

It's a little unclear from the question and comments whether you want to append to the lists, or append lists to the array. I've just demonstrated appending to the lists.

There is an np.append function, which new users often misuse. It isn't a substitute for list append. It is a front end to np.concatenate. It is not an in-place operation; it returns a new array.

Also defining a list to add with it can be tricky:

In [72]: np.append(A,[[1,23]])
Out[72]: array([[None, 0, 34], [None, 1, 34], [None, 2, 34], 1, 23],     dtype=object)

You need to construct another object array to concatenate to the original, e.g.

In [76]: np.append(A,np.empty((1,),dtype=object))
Out[76]: array([[None, 0, 34], [None, 1, 34], [None, 2, 34], None], dtype=object)

In all of this, an array of lists is harder to construct than a list of lists, and no easier, or faster, to manipulate. You have to make it a 2d array of lists to derive some benefit.

In [78]: A[:,None]
Out[78]: 
array([[[None, 0, 34]],
       [[None, 1, 34]],
       [[None, 2, 34]]], dtype=object)

You can reshape, transpose, etc an object array, where as creating and manipulating a list of lists of lists gets more complicated.

In [79]: A[:,None].tolist()
Out[79]: [[[None, 0, 34]], [[None, 1, 34]], [[None, 2, 34]]]

===

As shown in https://mcmap.net/q/280948/-how-to-initialize-numpy-array-of-list-objects, np.frompyfunc is a good tool for creating an array of objects.

np.frompyfunc(list, 0, 1)(np.empty((3,2), dtype=object))  

Counterattraction

After massaging my data, how can I convert this array of arrays back to a regular numpy array? (assuming the inner arrays are now all the same shape) – Glazing

@evn, some version of concatenate can rejoin arrays into one. np.stack() is often a good choice. Make sure the outer array is 1d, because they treat it like a list of arrays. – Counterattraction

Answer 2

If you really need a 1-d array of lists you will have to wrap your lists in your own class as numpy will always try to convert your lists to arrays inside of an array (which is more efficient but obviously requires constant size-elements), for example through

class mylist:

    def __init__(self, l):
        self.l=l

    def __repr__(self): 
        return repr(self.l)

    def append(self, x):
        self.l.append(x)

and then you can change any element without changing the dimension of others

>>> x = mylist([1,2,3])
>>> y = mylist([1,2,3])
>>> import numpy as np
>>> data = np.array([x,y])
>>> data
array([[1,2,3], [1,2,3]], dtype=object)
>>> data[0].append(2)
>>> data
array([[1,2,3,2], [1,2,3]], dtype=object)

Update

As suggested by ali_m there is actually a way to force numpy to simply create a 1-d array for references and then feed them with actual lists

>>> data = np.empty(2, dtype=np.object)
>>> data[:] = [1, 2, 3], [1, 2, 3]
>>> data
array([[1, 2, 3], [1, 2, 3]], dtype=object)
>>> data[0].append(4)
>>> data
array([[1, 2, 3, 4], [1, 2, 3]], dtype=object)

Estimative

In my case I don't know how many mylists I will have, how would it be in a generic way? – Chiachiack

You could achieve the same result without defining a new class using data = np.empty(2, dtype=np.object); data[:] = [1, 2, 3], [1, 2, 3] – Edify

@Ricardo if you do not know how many lists you have then why would you use arrays in the first place? They are constant size. If you want to add lists on the go, then probably a list of lists is preferable. Otherwise - you can always use np.concatenate to merge arrays – Estimative

An interesting observation @Estimative and a cautionary note to others looking for shortcuts... don't forget the data = np.empty(2, dtype=np.object) creation line. If you try to apparently skip a step by using ... data = np.array([[1,2,3],[4,5,6]],dtype=np.object) ... then try to ... data[0].append(4) it will return ... AttributeError: 'numpy.ndarray' object has no attribute 'append' in python 3.4.x . apparently an array of empty objects needs to be created first, followed by a fill. – Eblis

np.append is just an alternative way of calling np.concatenate. It is not a clone of list append. – Counterattraction

Well, I need to start a vector/array with N elements, each element will have a list of elements and they may vary size, and these lists I will append to. – Chiachiack

Answer 3

data = np.empty(20, dtype=np.object)
for i in range(data.shape[0]):
    data[i] = []
    data[i].append(i)
print(data)

The result will be:

[list([0]) list([1]) list([2]) list([3]) list([4]) list([5]) list([6]) list([7]) list([8]) list([9]) list([10]) list([11]) list([12]) list([13]) list([14]) list([15]) list([16]) list([17]) list([18]) list([19])]

Appertain

Answer 4

A simple way would be:

A = [[1,2],[3,4]] 
B = np.array(A+[[]])[:-1]

Perfective

Answer 5

Just found this, I've never answered a question before, but here is a pretty simple solution:

If you want a vector of length n, use:

A = np.array([[]]*n + [[1]])[:-1]

This returns:

array([list([]), list([]), ... , list([])], dtype=object)

If instead you want an n by m array, use:

A = np.array([[]]*n*m + [[1]])[:-1]
B = A.reshape((n,m))

For higher rank arrays, you can use a similar method by creating a long vector and reshaping it. This may not be the most efficient way, but it worked for me.

Cytochrome

Running this gives me an error that it is deprecated unless you add dtype=object – Vizza

[[]]*n makes a list of n references to the same empty list. – Counterattraction

Answer 6

Lists aren't very numpy anyway, so maybe a tuple of lists is good enough for you. You can get that easily and rather efficiently with an iterator expression:

fiveLists = tuple([] for _ in range(5))

You can leave out the tuple if you only need it once (gives you the raw iterator).

You can use this to create a numpy array if you really want to:

arrayOfLists = np.fromiter(([] for _ in range(5)), object)

Edit: as of July 2020, you get "ValueError: cannot create object arrays from iterator"

Alexander

Answer 7

if you need to create an array of the array from a sequence of lists or tuples

x=[[1,2],[3,4],[5,6]]
print(type(x))
print(type(x[0]))
#<class 'list'>
#<class 'list'>
import numpy as np
ar=np.array([np.array(i) for i in x],dtype=object)
print(type(ar))
print(type(ar[0]))
#<class 'numpy.ndarray'>
#<class 'numpy.ndarray'>

Sunderance

Answer 8

I realize this is a bit of a workaround if you don't need Pandas but it achieves the stated objective:

import pandas as pd

A = pd.Series([[1, 2], [3, 4]]).to_numpy()
assert isinstance(A[0], list)

Isis

Answer 9

Numpy array() does support a ndmin argument which allows you to set the minumim number of dimensions in the output array, but unfortunately does not (yet) support a ndmax argument which would allow this to happen easily.

In the meantime, here is a small function that will create a 1D array from an arbitrarily nested sequence:

def create_1d_array(seq: Sequence) -> np.ndarray:
    arr = np.empty(len(seq), dtype=object)
    arr[:] = [s for s in seq]
    return arr

>>> create_1d_array([[1, 2], [3, 4]])
array([list([1, 2]), list([3, 4])], dtype=object)

Alkanet

Answer 10

I had the same problem, elements of lists were added to the array as separate elements, not as lists. With help of @hpaulj I solved this problem as simple as:

array_of_lists = np.array(np.empty(1, dtype=object))
array_of_lists[0] = first_list
if second_list:
     array_of_lists = np.append(array_of_lists, np.empty(1, dtype=object))
     array_of_lists[1] = second_list
if third_list:
     array_of_lists = np.append(array_of_lists, np.empty(1, dtype=object))
     array_of_lists[2] = third_list

Hope this can help someone.

Obellia