How to delete and shift values in a pandas df column

Question

I have a pandas df that I want to manipulate so it's ordered. So for the df below, I'd like ['I'] to be ordered. So values would read 10-50. I have 2 options to do this;

1) Try to delete values in Column ['G'] or ['H']. So if values are == X then delete.

2) Try to merge values in the same Columns when they are == X

import pandas as pd

d = pd.DataFrame({
        'J' : [10,'B','C','C',50],
        'I' : ['B',20,30,40,'C'],
        'H' : ['X','A','C','B','X'],         
        'G' : ['X', 'B', 'A','B','X'],                                 
        })

Output:

   G  H   I   J
0  X  X   B  10
1  B  A  20   B
2  A  C  30   C
3  B  B  40   C
4  X  X   C  50

Option 1 is we delete X from Column H and the intended output would be:

   G  H   I   J
0  X  B  10
1  B  A  20   B
2  A  C  30   C
3  B  B  40   C
4  X  C  50

Option 2 is we merge on X in Column G-H and the intended output would be:

   G   H   I   J
0  XX  B  10
1  B   A  20   B
2  A   C  30   C
3  B   B  40   C
4  XX  C  50

I have played around with df = df.drop(df.H == 'X') but this deletes the whole row.

Christchurch

Can you add more data to DataFrame if necessary? If X is in H column there is always X in G column? – Pedlar

Answer 1

Option 1:

You can use shift the values to the left for rows that meet the criteria df.H == 'X':

With the following variable definitions:

hij = ['H', 'I', 'J']
x = df.H=='X'

We can write the shift assignment concisely.

df.loc[x, hij] = df.loc[x, hij].apply(lambda x: x.shift(-1), axis=1)
outputs:
    G   H   I   J
0   X   B   10  NaN
1   B   A   20  B
2   A   C   30  C
3   B   B   40  C
4   X   C   50  NaN

Option 2:

Same principle, but two statements are needed.

we can concat H to G

df.loc[x, 'G'] = df.loc[x, 'G'] + df.loc[x, 'H']
# df.loc[x, 'G'] = df.loc[x, ['G, 'H']].sum(axis=1)
# or df.loc[x, ['G', 'H']].apply(np.sum, axis=1)
# or df.loc[x, 'G'] = df.loc[x, ['G', 'H']].apply(lambda x: (x + x.shift(-1))[0], axis=1)

and shift as in option 1

df.loc[x, hij] = df.loc[x, hij].apply(lambda x: x.shift(-1), axis=1)
final output:
    G   H   I   J
0   XX  B   10  NaN
1   B   A   20  B
2   A   C   30  C
3   B   B   40  C
4   XX  C   50  NaN

Get

Answer 2

More general solution should be change condition if in G or H columns is X, then merge together and shift by condition:

d = pd.DataFrame({
        'J' : [10,'B','C','C',50, 60],
        'I' : ['B',20,30,40,'C', 'D'],
        'H' : ['X','A','C','B','X', 'Y'],         
        'G' : ['Y', 'B', 'A','B','X', 'X'],                                 
        }, columns=list('GHIJ'))
print (d)
   G  H   I   J
0  Y  X   B  10
1  B  A  20   B
2  A  C  30   C
3  B  B  40   C
4  X  X   C  50
5  X  Y   D  60

---

m = d[['G','H']].eq('X').any(axis=1)
print (m)

0     True
1    False
2    False
3    False
4     True
5     True
dtype: bool

d['H'] = d['G'] + d['H'] 
d[m] = d[m].shift(-1, axis=1)
print (d)
    G   H   I    J
0  YX   B  10  NaN
1   B  BA  20    B
2   A  AC  30    C
3   B  BB  40    C
4  XX   C  50  NaN
5  XY   D  60  NaN 

Pedlar

Answer 3

For you question one , replace the 'X' as np.nan , then sorted the data frame (shift the value)

d.replace({'H':{'X':np.nan}}).apply(lambda x: sorted(x, key=pd.isnull),1).fillna('')
Out[234]:
   G  H   I  J
0  X  B  10
1  B  A  20  B
2  A  C  30  C
3  B  B  40  C
4  X  C  50

For question two: using np.where create the d.G firstly , then we do the same as above

d.G=np.where((d.G=='X')&(d.H=='X'),'XX',d.G)

d.replace({'H':{'X':np.nan}}).apply(lambda x: sorted(x, key=pd.isnull),1).fillna('')


Out[242]: 
    G  H   I  J
0  XX  B  10   
1   B  A  20  B
2   A  C  30  C
3   B  B  40  C
4  XX  C  50   

Tremaine

Answer 4

You can go for np.where and shift i.e

ndf = pd.DataFrame(np.where((d['H']=='X')[:,None],
                       d.assign(H=d.H+d.G).shift(-1,axis=1), #only d.shift(...) in case you dont want to add  
                       d), columns=d.columns)

    G  H   I    J
0  XX  B  10  NaN
1   B  A  20    B
2   A  C  30    C
3   B  B  40    C
4  XX  C  50  NaN

Gypsum