Pandas .agg() convert to list but skip nans

Question

How do I consolidate/reduce a DataFrame so that it merges rows by custom column 'id' and puts values into a list if they are not Nan. So far I came up with this but it doesn't remove Nans:

x: pd.DataFrame = df_chunk.groupby('id', dropna=True).agg(lambda x: list(x))
for row in x.itertuples():
    print(row)

So the result is:

Pandas(Index=1, surname=['Bruce', nan, nan], given_name=['Erin', nan, nan], date_of_birth=['11/03/1961', '11/04/1961', '11/06/1961'], address=['10 Kestrel Wood Way, York', '4 Ward Avenue, Cowes', '11 Woodhill Court, Woodside Road, Amersham'], postcode=['YO31 9EJ', 'BD10 0LT', 'WA14 1LH'], mobile=['[phone]', '[phone]', '[phone]'])

Desired result is to have surname=['Bruce'], given_name=['Erin'] and so on

Mcshane

Answer 1

Use Series.dropna for remove NaNs and Nones:

df_chunk.groupby('id').agg(lambda x: list(x.dropna()))

Expiation