python del not freeing all the memory

Question

In my python program, I use pandas to read a csv file and store in memory:

data = pandas.read_csv('data.csv')

Before running the above command I check the free memory with free -m and the output is 1704. After running the above command the output is 729. I run

del(data)

to free the memory used by data. Now when I check the free memory the output is 1093 which is much less than the original 1704. Where did the rest go? How can I free it? I'm running all these in ipython and even exiting ipython doesn't free up that memory.

Thanks.

Gollin

You really shouldn't be worrying about memory management in Python. If you want that variable to go away do everything with it in a function scope. – Adolescent

Answer 1

"Exiting ipython doesn't free up that memory" means that you're seeing the effect at the OS level. You're probably seeing the effect of memory caching. That data is saved in memory until the space is needed or the file is read again - instead of reading the file from disk it will be read from the 'inactive' section of memory.

Assuming you're on some unix/linux variant, for details about ram usage

cat /proc/meminfo

Your available memory is Free + Inactive. The Free is available now, and the OS will reclaim space from Inactive as necessary. I keep the following alias (in csh) around to check

alias 'freemem' cat /proc/meminfo | grep -i 'inactive:\|memfree:'

Python's del will remove your variable from the namespace. So if that memory isn't referenced anywhere else in your code, python's garbage collection will kick in and clear data from memory. Or if you want to force it:

import gc
foo = range(10000000)
del foo
gc.collect()

Regardless of all this, the OS will keep the file contents cached so it doesn't have to do redundant disk reads when you try to read that same file again.

Timberlake

"Python's del will clear the processed data from memory" I don't think this is true. It won't clear the memory. It will remove it from the namespace and (I believe) set the reference count to 0. – Adolescent

Correct me if I'm wrong, but won't removing it from the namespace mark it as eligible for garbage collection, effectively clearing it from memory? on my system with (OSX, Python 2.7.10 |Anaconda 2.3.0 (x86_64)| ) I can make 800 MB worth of numpy arrays, see the effect in top, use del, and the memory is freed. – Timberlake

Maybe I'm just being picky, because you're not technically wrong, but "marking it for garbage collection" isn't the same as freeing the memory in my mind. You're marking the memory to be freed by the garbage collector. You aren't guaranteed that that memory is freed by the next line of Python code, but you are guaranteed you can't try to access it anymore and that it will go away on the next garbage collection. – Adolescent

I guess I run a little fast and loose. You're definitely right. I'll edit my answer to reflect what's really going on, rather than what's "effectively" going on. – Timberlake

I upvoted you anyway because I think your comment about caching was more of an actual answer to what is going on than mine :) – Adolescent

Thanks for the upvote, and for the semantics. It's important to be clear about the distinction here. – Timberlake

Answer 2

del is not C's free or C++'s delete

6.5. The del statement

del_stmt ::= "del" target_list

Deletion is recursively defined very similar to the way assignment is defined. Rather than spelling it out in full details, here are some hints.

Deletion of a target list recursively deletes each target, from left to right.

Deletion of a name removes the binding of that name from the local or global namespace, depending on whether the name occurs in a global statement in the same code block. If the name is unbound, a NameError exception will be raised.

It is illegal to delete a name from the local namespace if it occurs as a free variable in a nested block.

Deletion of attribute references, subscriptions and slicings is passed to the primary object involved; deletion of a slicing is in general equivalent to assignment of an empty slice of the right type (but even this is determined by the sliced object).

source Python spec

You're not freeing the memory. You're telling python you're done with that variable.

Adolescent

Thanks. Now what's a good way to free the whole memory that was used by that variable? – Gollin

There really isn't a good way to truly do that in Python. You don't get control of memory. All you can do is let the variable leave scope or use del to mark it for garbage collection. Also note @scf's answer in regards to caching. If you're writing code that requires that type of control, you're using the wrong language. You can write only that function in C or you can switch entirely to C[++] or some other lower level language. – Adolescent

@Gollin I just came across this: docs.python.org/2/library/gc.html If you use del as above, you can use gc.collect() to force garbage collection. – Timberlake