Calculating memory fragmentation in Python

Question

I have a long running process that allocates and releases objects constantly. Although objects are being freed, the RSS mem usage goes up over time.

How can I calculate how much fragmentation is happening? One possibility is to calculate RSS / sum_of_allocations and take that as an indicator. even then, how to do I calculate the denominator (sum_of_allocations).

Zoe

why do you think it is a memory fragmentation issue? Try Python memory profiler – Haemocyte

@J.F.Sebastian I was going to do that next. But Python is known to fragment memory for long lived processes that allocate/release small chunks of memory. – Zoe

@J.F.Sebastian So using the profiler, how do you figure if fragmentation is happening or not ? Do you just sum the memory allocations and compare that with RSS ? – Zoe

Memory profiler allows you to find a memory leak (an alternative explanation to "RSS mem usage goes up"). – Haemocyte

Did you manage to solve this in the end? I'm in a similar boat. – Emarie

Answer 1

Check out the Garbage Collector interface, gc.

http://docs.python.org/2/library/gc.html

You can inspect the objects are being tracked with gc.get_objects()

"As a general rule, instances of atomic types aren’t tracked and instances of non-atomic types (containers, user-defined objects...) are."

There is also gc.garbage, which finds objects that can't be freed but are unreachable.

Incidentally