Pandas: Chained assignments [duplicate]

Question

I have been reading this link on "Returning a view versus a copy". I do not really get how the chained assignment concept in Pandas works and how the usage of .ix(), .iloc(), or .loc() affects it.

I get the SettingWithCopyWarning warnings for the following lines of code, where data is a Panda dataframe and amount is a column (Series) name in that dataframe:

data['amount'] = data['amount'].astype(float)

data["amount"].fillna(data.groupby("num")["amount"].transform("mean"), inplace=True)

data["amount"].fillna(mean_avg, inplace=True)

Looking at this code, is it obvious that I am doing something suboptimal? If so, can you let me know the replacement code lines?

I am aware of the below warning and like to think that the warnings in my case are false positives:

The chained assignment warnings / exceptions are aiming to inform the user of a possibly invalid assignment. There may be false positives; situations where a chained assignment is inadvertantly reported.

EDIT : the code leading to the first copy warning error.

data['amount'] = data.apply(lambda row: function1(row,date,qty), axis=1) 
data['amount'] = data['amount'].astype(float)

def function1(row,date,qty):
    try:
        if(row['currency'] == 'A'):
            result = row[qty]
        else:
            rate = lookup[lookup['Date']==row[date]][row['currency'] ]
            result = float(rate) * float(row[qty])
        return result
    except ValueError: # generic exception clause
        print "The current row causes an exception:"

Imperator

Answer 1

The point of the SettingWithCopy is to warn the user that you may be doing something that will not update the original data frame as one might expect.

Here, data is a dataframe, possibly of a single dtype (or not). You are then taking a reference to this data['amount'] which is a Series, and updating it. This probably works in your case because you are returning the same dtype of data as existed.

However it could create a copy which updates a copy of data['amount'] which you would not see; Then you would be wondering why it is not updating.

Pandas returns a copy of an object in almost all method calls. The inplace operations are a convience operation which work, but in general are not clear that data is being modified and could potentially work on copies.

Much more clear to do this:

data['amount'] = data["amount"].fillna(data.groupby("num")["amount"].transform("mean"))

data["amount"] = data['amount'].fillna(mean_avg)

One further plus to working on copies. You can chain operations, this is not possible with inplace ones.

e.g.

data['amount'] = data['amount'].fillna(mean_avg)*2

And just an FYI. inplace operations are neither faster nor more memory efficient. my2c they should be banned. But too late on that API.

You can of course turn this off:

pd.set_option('chained_assignment',None)

Pandas runs with the entire test suite with this set to raise (so we know if chaining is happening) on, FYI.

Faintheart

Thanks Jeff, so I should ideally remove the inplace parameters for the 2nd and 3rd warnings. Regarding the 1st one, i.e. data['amount'] = data['amount'].astype(float), what would be a replacement that does not produce the copy warning? – Imperator

you must be doing something before the astype assignment. can you show more code? – Faintheart

sure, I added the code to my question. – Imperator

can you show data.info() before this? you should have float64 dtypes already. secondarily, you don't need the apply, you can do something like: data[data['currency']!='A','amount']=data['qty']*data['rate'] – Faintheart

Thanks @Faintheart for this solution: pd.set_option('chained_assignment',None) However I'm wondering if this is recommended since I'm always conservative in changing default warning settings... – Occur