I am trying two different lines of code that both involve computing combinations of rows of a df with 500k rows.
I think bc of the large # of combinations, the kernal keeps dying. Is there anyway to resolve this ?
Both lines of code that crash are
pd.merge(df.assign(key=0), df.assign(key=0), on='key').drop('key', axis=1)
and
index_comb = list(combinations(df.index, 2))
Both are different ways to achieve same desired df but kernal fails on both.
Would appreciate any help :/
Update: I tried using the code in my terminal and it gives me an error of killed 9: it is using too much memory in terminal as well?