How to find the line that is generating a Pandas SettingWithCopyWarning?

Question

I have a large block of code that is, at some point somewhere, generating a setting with copy warning in pandas (this problem).

I know how to fix the problem, but I can't find what line number it is! Is there a way to back out the line number (apart from brute force methods like debug-stepping or putting in multiple prints)? The only output I get is the below, which doesn't go up the stack to my code:

C:\Anaconda3\lib\site-packages\pandas\core\frame.py:2302: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame  **kwargs)

Pibroch

How large is the code? I mean can't you just scan the code for =? – Senescent

You could divide and conquer, comment out the last half of your code and repeat until the warning goes away then you'll find the line, you could have more warnings later on but it'll probably trigger a pattern for you to recognise. I'd look for things where you take a reference to some rows or a df column and then try to assign to it – Senescent

I'd say you use your IDE's debugger, set a breakpoint at the given line and look at the stack trace. If you don't have an IDE, read about Python's internal debugger at docs.python.org/2/library/pdb.html – Shiftless

Thanks Klaus - your tip did it. Placing a breakpoint at line 1182 of pandas\core\generic.py (where the warning was generated from, which I found by full-text searching the pandas/core directory), and then moving up the stack got me to the line number I need. – Pibroch

Apparently pandas now tells you the line number in more recent versions (> .14 I think) – Pibroch

This answer to my similar question may be helpful too. – Hillis

Answer 1

Set pd.options.mode.chained_assignment = 'raise'

This will throw an exception pointing to the line which triggers SettingWithCopyError.

UPDATE: how to catch the error, and interrogate the stacktrace to get the actual offending lineno:

import pandas as pd
from inspect import currentframe, getframeinfo
from pandas.core.common import SettingWithCopyError

pd.options.mode.chained_assignment = 'raise'

df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})

df2 = df[df['a'] == 2]

try:
    df2['b'] = 'foo'
except SettingWithCopyError:
    print('handling..')
    frameinfo = getframeinfo(currentframe())
    print(frameinfo.lineno)

Crevasse

Can the error be raised but the code runs further? I mean only to show the number of lines of code, where the warning(s) is/are happening. – Georgy

Yes, you just need to catch the error. As for the line number you can either add it manually using logging/print statements, or extract it using inspect module. Please see the update in the answer. – Crevasse

If you want to change it back after finding the line, you could use pd.options.mode.chained_assignment = 'warn' – Lazar