Memory use of pickling/unpickling enum in Python

While investigating the memory use of an application that exchanges message through PyZMQ using tracemalloc, I noticed a weird behavior of pickling and unpickling enums: memory looks like it is leaking.

This seems impossible to me so I am guessing I am measuring something in an incorrect manner. Could someone let me know if this a real leak or, if it is not, how I should be measuring memory usage to avoid seeing such behavior ?

I reduced my test to:

import enum
import gc
import os
import pickle
import tracemalloc
from uuid import SafeUUID


class A:
    pass


@enum.unique
class E(enum.Enum):
    A = 1


obj = A()
tracemalloc.start()
for obj in (A(), E.A, SafeUUID.safe):
    print(obj)
    for j in range(10):
        tracemalloc.clear_traces()
        i = 0
        while True:
            s = pickle.dumps(obj)
            o = pickle.loads(s)

            del s, o

            if i == 100:
                gc.collect()
                origin = tracemalloc.take_snapshot().filter_traces(
                    [tracemalloc.Filter(True, __file__)]
                )
            elif i > 200:
                gc.collect()
                snap = tracemalloc.take_snapshot().filter_traces(
                    [tracemalloc.Filter(True, __file__)]
                )
                print(str(snap.compare_to(origin, "lineno")[0]).rsplit(os.sep, 1)[-1])
                break
            i += 1
    print()

The exact output is noisy but I can get things similar to the following, in which pickling a class instance displays no variation, pickling a local enum see some memory increase, and pickling an enum defined in a different module can result in large memory swings.

<__main__.A object at 0x000002840A56B150>
enum-pickle-leak.py:27: size=1520 B (+1520 B), count=1 (+1), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B
enum-pickle-leak.py:27: size=1520 B (+0 B), count=1 (+0), average=1520 B

E.A
enum-pickle-leak.py:26: size=3640 B (+112 B), count=65 (+2), average=56 B
enum-pickle-leak.py:27: size=4040 B (+168 B), count=46 (+3), average=88 B
enum-pickle-leak.py:27: size=7288 B (+224 B), count=104 (+4), average=70 B
enum-pickle-leak.py:27: size=2528 B (+168 B), count=19 (+3), average=133 B
enum-pickle-leak.py:26: size=392 B (+112 B), count=7 (+2), average=56 B
enum-pickle-leak.py:26: size=168 B (+112 B), count=3 (+2), average=56 B
enum-pickle-leak.py:27: size=4600 B (+112 B), count=56 (+2), average=82 B
enum-pickle-leak.py:27: size=1800 B (+168 B), count=6 (+3), average=300 B
enum-pickle-leak.py:27: size=1744 B (+112 B), count=5 (+2), average=349 B
enum-pickle-leak.py:27: size=3312 B (+112 B), count=33 (+2), average=100 B

SafeUUID.safe
enum-pickle-leak.py:27: size=16.6 KiB (+10.3 KiB), count=283 (+193), average=60 B
enum-pickle-leak.py:27: size=12.9 KiB (+1098 B), count=214 (+20), average=62 B
enum-pickle-leak.py:27: size=12.3 KiB (+494 B), count=202 (+9), average=62 B
enum-pickle-leak.py:27: size=5539 B (+440 B), count=74 (+8), average=75 B
enum-pickle-leak.py:27: size=2678 B (+659 B), count=22 (+12), average=122 B
enum-pickle-leak.py:27: size=3343 B (+887 B), count=34 (+16), average=98 B
enum-pickle-leak.py:27: size=3377 B (+599 B), count=35 (+11), average=96 B
enum-pickle-leak.py:27: size=9367 B (+551 B), count=144 (+10), average=65 B
enum-pickle-leak.py:27: size=2834 B (+434 B), count=25 (+8), average=113 B
enum-pickle-leak.py:27: size=3005 B (+771 B), count=28 (+14), average=107 B

Recommended topics

Hot tags