Efficient method to calculate the rank vector of a list in Python, handle ties

Question

I'm looking for an efficient way to calculate the rank vector of a list in Python, similar to R's rank function. In a simple list with no ties between the elements, element i of the rank vector of a list l should be x if and only if l[i] is the x-th element in the sorted list. This is simple so far, the following code snippet does the trick:

def rank_simple(vector):
    return sorted(range(len(vector)), key=vector.__getitem__)

Things get complicated, however, if the original list has ties (i.e. multiple elements with the same value). In that case, all the elements having the same value should have the same rank, which is the average of their ranks obtained using the naive method above. So, for instance, if I have [1, 2, 3, 3, 3, 4, 5], the naive ranking gives me [0, 1, 2, 3, 4, 5, 6], but what I would like to have is [0, 1, 3, 3, 3, 5, 6]. Which one would be the most efficient way to do this in Python?

---

Footnote: let me know if there's a NumPy method to achieve this; but I am interested in a pure Python solution anyway as I'm developing a tool which should work without NumPy as well.

Steinman

have you checked numpy.argsort(vector) ? – Copyread

BTW, I think, this code can not even calculate ordinal rank. To calculate ordinal rank correctly, read this – Shaven

Almost duplicate of Rank items in an array using Python/NumPy, without sorting array twice - Stack Overflow -- except that the other question explicitly asks for a numpy solution. – Footie

Sorry for disturbing after eleven years, but... Isn't your rank_simple() actually the equivalent of R's order() function, instead of rank()? See e.g. https://mcmap.net/q/103427/-rank-and-order-in-r. – Muleteer

You don't need the tie-breaker to be 'average'; it could be 'min'/'max'; R's rank() fucntion allow you to specify any of those, as well as 'first', 'last', 'random'. – Loudermilk

Answer 1

Using scipy, the function you are looking for is scipy.stats.rankdata:

In [13]: import scipy.stats as ss
In [19]: ss.rankdata([3, 1, 4, 15, 92])
Out[19]: array([ 2.,  1.,  3.,  4.,  5.])

In [20]: ss.rankdata([1, 2, 3, 3, 3, 4, 5])
Out[20]: array([ 1.,  2.,  4.,  4.,  4.,  6.,  7.])

The ranks start at 1, rather than 0 (as in your example), but then again, that's the way R's rank function works as well.

Here is a pure-python equivalent of scipy's rankdata function:

def rank_simple(vector):
    return sorted(range(len(vector)), key=vector.__getitem__)

def rankdata(a):
    n = len(a)
    ivec=rank_simple(a)
    svec=[a[rank] for rank in ivec]
    sumranks = 0
    dupcount = 0
    newarray = [0]*n
    for i in xrange(n):
        sumranks += i
        dupcount += 1
        if i==n-1 or svec[i] != svec[i+1]:
            averank = sumranks / float(dupcount) + 1
            for j in xrange(i-dupcount+1,i+1):
                newarray[ivec[j]] = averank
            sumranks = 0
            dupcount = 0
    return newarray

print(rankdata([3, 1, 4, 15, 92]))
# [2.0, 1.0, 3.0, 4.0, 5.0]
print(rankdata([1, 2, 3, 3, 3, 4, 5]))
# [1.0, 2.0, 4.0, 4.0, 4.0, 6.0, 7.0]

Shelves

removed x from xrange so just range and this worked well. not sure if xrange was a typo or if I'm missing something, but thanks! – Howler

@Howler xrange is a built-in function from Python 2 that no longer exists in Python 3. Using range like you did is the correct approach now. – Scrummage

Answer 2

[sorted(l).index(x) for x in l]

sorted(l) will give the sorted version index(x) will give the index in the sorted array

for example :

l = [-1, 3, 2, 0,0]
>>> [sorted(l).index(x) for x in l]
[0, 4, 3, 1, 1]

Yulandayule

Nice one-liner! Considering efficiency, does this repeat sorting for every x in l? BTW, it returns the lowest index for tied ranks, not the average, which is another useful option but not exactly what the OP asked. – Goddamn

Answer 3

This is one of the functions that I wrote to calculate rank.

def calculate_rank(vector):
  a={}
  rank=1
  for num in sorted(vector):
    if num not in a:
      a[num]=rank
      rank=rank+1
  return[a[i] for i in vector]

input:

calculate_rank([1,3,4,8,7,5,4,6])

output:

[1, 2, 3, 7, 6, 4, 3, 5]

Craftsman

Answer 4

This doesn't give the exact result you specify, but perhaps it would be useful anyways. The following snippet gives the first index for each element, yielding a final rank vector of [0, 1, 2, 2, 2, 5, 6]

def rank_index(vector):
    return [vector.index(x) for x in sorted(range(n), key=vector.__getitem__)]

Your own testing would have to prove the efficiency of this.

Humour

This assumes that vector is already sorted, but still a very understandable implementation. +1 – Convivial

Ah, good point. Tamás's comprehension begins with a sorted() list... I'll edit to include that. – Humour

not only the assumption does not hold, but also the index() method is O(N) as well, so not efficient at all. – Furtive

Answer 5

Here is a small variation of unutbu's code, including an optional 'method' argument for the type of value of tied ranks.

def rank_simple(vector):
    return sorted(range(len(vector)), key=vector.__getitem__)

def rankdata(a, method='average'):
    n = len(a)
    ivec=rank_simple(a)
    svec=[a[rank] for rank in ivec]
    sumranks = 0
    dupcount = 0
    newarray = [0]*n
    for i in xrange(n):
        sumranks += i
        dupcount += 1
        if i==n-1 or svec[i] != svec[i+1]:
            for j in xrange(i-dupcount+1,i+1):
                if method=='average':
                    averank = sumranks / float(dupcount) + 1
                    newarray[ivec[j]] = averank
                elif method=='max':
                    newarray[ivec[j]] = i+1
                elif method=='min':
                    newarray[ivec[j]] = i+1 -dupcount+1
                else:
                    raise NameError('Unsupported method')

            sumranks = 0
            dupcount = 0


    return newarray

Elsewhere

Thank you! Recent versions of scipy.stats.rankdata have the optional method argument, but I'm stuck working with an older version that only supports the average method, so you saved me a lot of time writing my own function. If you add a "dense" option then you'll have covered it all. – Segregation

Answer 6

I really don't get why all the existing solutions are so complex. This can be done just like this:

[index for element, index in sorted(zip(sequence, range(len(sequence))))]

You build tuples which contain the elements and a running index. Then you sort the whole thing, and tuples sort by their first element and during ties by their second element. This way one has a sorted list of these tuples and just need to pick out the indices from that afterwards. Also this removes the need to look up elements in the sequence afterwards, which likely makes it a O(N²) operation whereas this is O(N log(N)).

Mcwhorter

Since tuple sorting will sort by the second element when the first two are equal, ties will be numbered ascendingly. – Mcwhorter

Its nice solution for ranking ties but OP asked for "all the elements having the same value should have the same rank" – Appreciation

@amonowy: Finally I understand why the other solutions are so complex. Then this answer does not match the question. Shall I delete it? – Mcwhorter

For me it was beneficial to analyze it and I think its worth to keep it. – Appreciation

Answer 7

There is a really nice module called Ranking http://pythonhosted.org/ranking/ with an easy to follow instruction page. To download, simply use easy_install ranking

Bottomry

Answer 8

So.. this is 2019, and I have no idea why nobody suggested the following:

# Python-only
def rank_list( x, break_ties=False ):
    n = len(x)
    t = list(range(n))
    s = sorted( t, key=x.__getitem__ )

    if not break_ties:
        for k in range(n-1):
            t[k+1] = t[k] + (x[s[k+1]] != x[s[k]])

    r = s.copy()
    for i,k in enumerate(s):
        r[k] = t[i]

    return r

# Using Numpy, see also: np.argsort
def rank_vec( x, break_ties=False ):
    n = len(x)
    t = np.arange(n)
    s = sorted( t, key=x.__getitem__ )

    if not break_ties:
        t[1:] = np.cumsum(x[s[1:]] != x[s[:-1]])

    r = t.copy()
    np.put( r, s, t )
    return r

This approach has linear runtime complexity after the initial sort, it only stores 2 arrays of indices, and does not require values to be hashable (only pairwise comparison needed).

AFAICT, this is better than other approaches suggested so far:

• @unutbu's approach is essentially similar, but (I would argue) too complicated for what the OP asked;
• All suggestions using .index() are terrible, with a runtime complexity of N^2;
• @Yuvraj Singh improves slightly upon the .index() search using a dictionary, however with search and insert operations at each iteration, this is still highly inefficient both in time (NlogN) and space, and it also requires the values to be hashable.

Taut

Answer 9

The most pythonic style to find rank of an array:

a = [10.0, 9.8, 8.0, 7.8, 7.7, 7.0, 6.0, 5.0, 4.0, 2.0]
rank = lambda arr: list(map(lambda i: sorted(arr).index(i)+1, arr))
rank(a)

Vienne

Answer 10

These codes give me a lot of inspiration, especially unutbu's code. However my needs are simpler, so I changed the code a little.

Hoping to help the guys with the same needs.

Here is the class to record the players' scores and ranks.

class Player():
    def __init__(self, s, r):
        self.score = s
        self.rank = r

Some data.

l = [Player(90,0),Player(95,0),Player(85,0), Player(90,0),Player(95,0)]

Here is the code for calculation:

l.sort(key=lambda x:x.score, reverse=True)    
l[0].rank = 1
dupcount = 0
prev = l[0]
for e in l[1:]:
    if e.score == prev.score:
        e.rank = prev.rank
        dupcount += 1
    else:
        e.rank = prev.rank + dupcount + 1
        dupcount = 0
        prev = e

Antofagasta

Answer 11

import numpy as np

def rankVec(arg):
    p = np.unique(arg) #take unique value
    k = (-p).argsort().argsort() #sort based on arguments in ascending order
    dd = defaultdict(int)
    for i in xrange(np.shape(p)[0]):
        dd[p[i]] = k[i]
    return np.array([dd[x] for x in arg])

timecomplexity is 46.2us

Brockbrocken

While this code may answer the question, providing additional context regarding how and/or why it solves the problem would improve its long-term value. See this – Taut

Answer 12

This works for the spearman correlation coefficient .

def get_rank(X, n):
    x_rank = dict((x, i+1) for i, x in enumerate(sorted(set(X))))
    return [x_rank[x] for x in X]

Ultravirus

Answer 13

The rank function could be implemented in O(n log n) time and O(n) additional space using the following approach.

import bisect

def rank_list(lst: list[int]) -> list[int]:
    sorted_vals = sorted(set(lst))
    return [bisect.bisect_left(sorted_vals, val) for val in lst]

I use here bisect library, but for the pure independent code it is enough to implement binary search procedure on the sorted array with unique values for a query on existing (in this array) value.

Discomfort