how do you find and save duplicated rows in a numpy array?

Question

I have an array e.g.

Array = [[1,1,1],[2,2,2],[3,3,3],[4,4,4],[5,5,5],[1,1,1],[2,2,2]]

And i would like something that would output the following:

Repeated = [[1,1,1],[2,2,2]]

Preserving the number of repeated rows would work too, e.g.

Repeated = [[1,1,1],[1,1,1],[2,2,2],[2,2,2]]

I thought the solution might include numpy.unique, but i can't get it to work, is there a native python / numpy function?

Indebtedness

But unique is used to obtain the unique numbers, not the duplicate ones. Is this always a list of list of integers? Or can the objects be arbitrary objects? – Jeanejeanelle

Did either of the posted solutions work for you? – Blush

Answer 1

Using the new axis functionality of np.unique alongwith return_counts=True that gives us the unique rows and the corresponding counts for each of those rows, we can mask out the rows with counts > 1 and thus have our desired output, like so -

In [688]: a = np.array([[1,1,1],[2,2,2],[3,3,3],[4,4,4],[5,5,5],[1,1,1],[2,2,2]])

In [689]: unq, count = np.unique(a, axis=0, return_counts=True)

In [690]: unq[count>1]
Out[690]: 
array([[1, 1, 1],
       [2, 2, 2]])

Blush

Is it possible to get the indexes of repeated rows? For example, [0, 5] and [1, 2]. – Dominic

Answer 2

If you need to get indices of the repeated rows

import numpy as np

a = np.array([[1,1,1],[2,2,2],[3,3,3],[4,4,4],[5,5,5],[1,1,1],[2,2,2]])
unq, count = np.unique(a, axis=0, return_counts=True)
repeated_groups = unq[count > 1]

for repeated_group in repeated_groups:
    repeated_idx = np.argwhere(np.all(a == repeated_group, axis=1))
    print(repeated_idx.ravel())

# [0 5]
# [1 6]

Hyacinthie

Answer 3

You could use something like Repeated = list(set(map(tuple, Array))) if you didn't necessarily need order preserved. The advantage of this is you don't need additional dependencies like numpy. Depending on what you're doing next, you could probably get away with Repeated = set(map(tuple, Array)) and avoid a type conversion if you would like.

Lejeune