AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>' when implementing Pytorch framework

Question

I was trying to implement a pytorch framework on CNN.
I'm sure the code is right because it's from a tutorial and it works when I ran it on Jupyter Notebook on GoogleDrive.
But when I tried to localize it as a .py file, it suggest an error:
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>' I know it's about inferencing objects outside a function, but what was the exact matter about this error?
And how should I fix it?

Here's the major part of the code.

def pre_datasets():
    TRAIN_TFM = transforms.Compose(
        [
            transforms.Resize(size=(128, 128)),
            # TODO
            transforms.ToTensor(),
        ]
    )
    train_set = DatasetFolder(
        root=CONFIG["train_set_path"],
        loader=lambda x: Image.open(x),
        extensions="jpg",
        transform=TRAIN_TFM,
    )
    train_loader = DataLoader(
        dataset=train_set,
        batch_size=CONFIG["batch_size"],
        shuffle=True,
        num_workers=CONFIG["num_workers"],
        pin_memory=True,
    )
    return train_loader

def train(train_loader):
    ...
    for epoch in range(CONFIG["num_epochs"]):
    ...
        for batch in train_loader: # error happened here
    ...

if __name__ == "__main__":
    train_loader = pre_datasets()
    train(train_loader)

Here's the error message:

Traceback (most recent call last):
  File "HW03_byCRZ.py", line 197, in <module>
    train(train_loader, valid_loader)
  File "HW03_byCRZ.py", line 157, in train
    for batch in train_loader:
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 355, in __iter__
    return self._get_iterator()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 301, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 914, in __init__
    w.start()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'

Corbel

I've solved the problem. I avoid the problem by directly replacing the lambda x:Image.open(x) part with Image.open. – Corbel

Answer 1

I had a similar issue and I used dill like this:

import dill as pickle

and it worked out of the box!

Lath

Just to make this more clear, dill is serializing the lambda function out of the box where as pickle does not do this. This fixed it for me as well. – Metrical

Answer 2

Pickle'ing a lambda function requires extra modifications (serializing the function) see Can Python pickle lambda functions? for more information on how to do it.

Helfrich

I've solved the problem. I avoid the problem by directly replacing the lambda x:Image.open(x) part with Image.open. But the pickle'ing a function part helped me a lot about understanding what's going on. – Corbel

Oh thats nice to hear. Also have you tried adding import dill to the beginning of your code. It would possibly solve that issue. – Helfrich

Yeah, I tried that, but the console said that I had to do more changes. So I edit the code as the simpler way as I mentioned. – Corbel