Pandas: SettingWithCopyWarning [duplicate]

Question

I'd like to replace values in a Pandas DataFrame larger than an arbitrary number (100 in this case) with NaN (as values this large are indicative of a failed experiment). Previously I've used this to replace unwanted values:

sve2_all[sve2_all[' Hgtot ng/l'] > 100] = np.nan

However, I got the following error:

-c:3: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
C:\Users\AppData\Local\Enthought\Canopy32\User\lib\site-packages\pandas\core\indexing.py:346: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_index,col_indexer] = value instead
self.obj[item] = s

From this StackExchange question, it seems that sometimes this warning can be ignored, but I can't follow the discussion well enough to be certain whether this applies to my situation. Is the warning basically letting me know that I'll be overwriting some of the values in my DataFrame?

Edit: As far as I can tell, everything behaved as it should. As a follow up is my method of replacing values non-standard? Is there a better way to replace values?

Cortie

Answer 1

As suggested in the error message, you should use loc to do this:

sve2_all.loc[sve2_all['Hgtot ng/l'] > 100] = np.nan

The warning is here to stop you modifying a copy (here sve2_all[sve2_all[' Hgtot ng/l'] > 100] is potentially a copy, and if it is then any modifications would not change the original frame. It could be that it works correctly in some cases but pandas cannot guarantee it will work in all cases... use at your own risk (consider yourself warned! ;) ).

Carnet

Answer 2

I was getting this warning while trying to reset the contents of an entire DataFrame but couldn't resolve it using loc or iloc:

df.loc[:, :] = new_values # SettingWithCopyWarning
df.iloc[:, :] = new_values # SettingWithCopyWarning

But resolving to the ndarray contained as data solved the problem:

df.values[:, :] = new_values # no warnings and desired behavior

Juridical

I used the same to replace a single column, for df.loc I got this warning too. – Latreshia

Answer 3

---Problem solved for me---

I had that warring error when i tried to convert float --> int even if i used the ".loc" command. my mistake was that i filtered my dataFrame (with masks) before the operation so the conversion occurred in only a small part of the dataframe item/column, the result was a mixed type column wich create a confuison. i solved the problem by converting the data frame before the masks (data filtration), i hope it will help.

Ipoh

Answer 4

As it is suggested by other users, you can try:

myindex = sve2_all[' Hgtot ng/l'] > 100

sve2_all.loc[myindex, 'yourcolumn'] = np.nan

Keep in mind that if you run into problems creating pivot tables (pivot_table row keyword not supported by pandas 0.16.0 #417) you should use the new syntax of index and columns instead of rows and cols. https://github.com/yhat/ggplot/issues/417

See also:

Pandas SettingWithCopyWarning

http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

Dutybound

You can check if there is True value in index and then apply replacement if (myindex==True).sum()>0: df_train.loc[myindex, c] = 0 – Dinka