Pandas Equivalent of R's which()

Question

Variations of this question have been asked before, I'm still having trouble understanding how to actually slice a python series/pandas dataframe based on conditions that I'd like to set.

In R, what I'm trying to do is:

df[which(df[,colnumber] > somenumberIchoose),]

The which() function finds indices of row entries in a column in the dataframe which are greater than somenumberIchoose, and returns this as a vector. Then, I slice the dataframe by using these row indices to indicate which rows of the dataframe I would like to look at in the new form.

Is there an equivalent way to do this in python? I've seen references to enumerate, which I don't fully understand after reading the documentation. My sample in order to get the row indices right now looks like this:

indexfuture = [ x.index(), x in enumerate(df['colname']) if x > yesterday]  

However, I keep on getting an invalid syntax error. I can hack a workaround by for looping through the values, and manually doing the search myself, but that seems extremely non-pythonic and inefficient.

What exactly does enumerate() do? What is the pythonic way of finding indices of values in a vector that fulfill desired parameters?

Note: I'm using Pandas for the dataframes

Leopold

Can you show the full traceback of the error? – Cathiecathleen

can you try: [a.index() for (a, b) in enumerate(df['colname']) if b > yesterday] – Neptune

Just to be clear, pandas DataFrames can have all sorts of indices, not just integers. Do you only want integer indices, or the actual original row-indices? – Recaption

Related question Python equivalent of which() in R – Recaption

The question asks about which() which returns a vector of indices in which some condition was met. The top answer is about boolean subsetting. This post contains what I see as an actual equivalent to which(). – Rydder

Answer 1

I may not understand clearly the question, but it looks like the response is easier than what you think:

using pandas DataFrame:

df['colname'] > somenumberIchoose

returns a pandas series with True / False values and the original index of the DataFrame.

Then you can use that boolean series on the original DataFrame and get the subset you are looking for:

df[df['colname'] > somenumberIchoose]

should be enough.

See http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing

Prunella

df[df['colname'] > somenumberIchoose].index is the same as R which() function – Natty

Answer 2

What what I know of R you might be more comfortable working with numpy -- a scientific computing package similar to MATLAB.

If you want the indices of an array who values are divisible by two then the following would work.

arr = numpy.arange(10)
truth_table = arr % 2 == 0
indices = numpy.where(truth_table)
values = arr[indices]

It's also easy to work with multi-dimensional arrays

arr2d = arr.reshape(2,5)
col_indices = numpy.where(arr2d[col_index] % 2 == 0)
col_values = arr2d[col_index, col_indices]

Milesmilesian

+1 for a solution much closer to the R idiom. Also I don't like to turn everything into a pandas dataframe. – Macmillan

Answer 3

enumerate() returns an iterator that yields an (index, item) tuple in each iteration, so you can't (and don't need to) call .index() again.

Furthermore, your list comprehension syntax is wrong:

indexfuture = [(index, x) for (index, x) in enumerate(df['colname']) if x > yesterday]

Test case:

>>> [(index, x) for (index, x) in enumerate("abcdef") if x > "c"]
[(3, 'd'), (4, 'e'), (5, 'f')]

Of course, you don't need to unpack the tuple:

>>> [tup for tup in enumerate("abcdef") if tup[1] > "c"]
[(3, 'd'), (4, 'e'), (5, 'f')]

unless you're only interested in the indices, in which case you could do something like

>>> [index for (index, x) in enumerate("abcdef") if x > "c"]
[3, 4, 5]

Cameraman

So Python : enumerate :: Ruby : each_with_index? – Paramatta

There's no need to use enumerate(), it's inefficient, and even if there was a need, pd.DataFrame has iterrows() for exactly that purpose. – Recaption

Answer 4

And if you need an additional statement panda.Series allows you to do Operations between Series (+, -, /, , *).

Just multiplicate the indexes:

idx1 = df['lat'] == 49
idx2 = df['lng'] > 15 
idx = idx1 * idx2

new_df = df[idx] 

Brewton

Answer 5

Instead of enumerate, I usually just use .iteritems. This saves a .index(). Namely,

[k for k, v in (df['c'] > t).iteritems() if v]

Otherwise, one has to do

df[df['c'] > t].index()

This duplicates the typing of the data frame name, which can be very long and painful to type.

Dit

I think it's just df.index, an attribute, not a function. I get an error that 'Int64Index' object is not callable with index(). That said, both of these actually answer how one can do what which() does, so I like that! – Rydder

Answer 6

A nice simple and neat way of doing this is the following:

SlicedData1 = df[df.colname>somenumber]]

This can easily be extended to include other criteria, such as non-numeric data:

SlicedData2 = df[(df.colname1>somenumber & df.colname2=='24/08/2018')]

And so on...

Hydropathy