AttributeError: Can't pickle local object '<locals>.<lambda>' [duplicate]

Question

I am trying to pickle a nested dictionary which is created using:

collections.defaultdict(lambda: collections.defaultdict(int))

My simplified code goes like this:

class A:
  def funA(self):
    #create a dictionary and fill with values
    dictionary = collections.defaultdict(lambda: collections.defaultdict(int))
    ...
    #then pickle to save it
    pickle.dump(dictionary, f)

However it gives error:

AttributeError: Can't pickle local object 'A.funA.<locals>.<lambda>'

After I print dictionary it shows:

defaultdict(<function A.funA.<locals>.<lambda> at 0x7fd569dd07b8> {...}

I try to make the dictionary global within that function but the error is the same. I appreciate any solution or insight to this problem. Thanks!

Mease

cast your defaultdict to dict and it should hopefully work – Fromma

You probably don't need a lambda; see #68680306 and https://mcmap.net/q/591403/-pytorch-can-39-t-pickle-lambda. – Burdett

Answer 1

pickle records references to functions (module and function name), not the functions themselves. When unpickling, it will load the module and get the function by name. lambda creates anonymous function objects that don't have names and can't be found by the loader. The solution is to switch to a named function.

def create_int_defaultdict():
    return collections.defaultdict(int)

class A:
  def funA(self):
    #create a dictionary and fill with values
    dictionary = collections.defaultdict(create_int_defaultdict)
    ...
    #then pickle to save it
    pickle.dump(dictionary, f)

Stiffen

Answer 2

As @tdlaney explained, lambda creates an anonymous function which can't be pickled. The most concise solution is to replace lambda with partial (no new function needed):

from functools import partial

class A:
  def funA(self):
    #create a dictionary and fill with values
    dictionary = collections.defaultdict(partial(collections.defaultdict, int))
    ...
    #then pickle to save it
    pickle.dump(dictionary, f)

Carbine