How can I check the memory usage of objects in iPython?

Question

I am using iPython to run my code. I wonder if there is any module or command which would allow me to check the memory usage of an object. For instance:

In [1]: a = range(10000)
In [2]: %memusage a
Out[2]: 1MB

Something like %memusage <object> and return the memory used by the object.

Duplicate

Find out how much memory is being used by an object in Python

Whiz

Duplicate: stackoverflow.com/questions/33978/…, stackoverflow.com/questions/512893/… – Hangdog

Also related: stackoverflow.com/questions/135664/… – Puparium

Sorry. I just wanna ask if there is any implementation of this features in ipython, or module for that adding the "magic" function in ipython (Since I use that for testing a lot.) – Whiz

Possible duplicate of How do I determine the size of an object in Python? – Wallen

Answer 1

Unfortunately this is not possible, but there are a number of ways of approximating the answer:

1. for very simple objects (e.g. ints, strings, floats, doubles) which are represented more or less as simple C-language types you can simply calculate the number of bytes as with John Mulder's solution.

2. For more complex objects a good approximation is to serialize the object to a string using cPickle.dumps. The length of the string is a good approximation of the amount of memory required to store an object.

There is one big snag with solution 2, which is that objects usually contain references to other objects. For example a dict contains string-keys and other objects as values. Those other objects might be shared. Since pickle always tries to do a complete serialization of the object it will always over-estimate the amount of memory required to store an object.

Arrowhead

But if you pickle a list containing all root objects you're interested in, there will be no overestimation. – Puparium

Thank you very much. But I wonder if pickle will do any compression or not. – Whiz

No, pickle does not compress. It simply eliminates redundancy. – Arrowhead

There is no 'above' solution on SO. – Sverige

Salim: Eliminating redundancy is one definition of compression :) I recently tried creating an array of 10,000 similar strings, and using Pickle to look at the memory consumption. It stores it as the string once, and then one byte for each repetition. However, this tells me nothing about the internal memory representation of this string (it might be the same, or it might be different). – Bushel

Answer 2

If you are using a numpy array, then you can use the attribute ndarray.nbytes to evaluate its size in memory:

from pylab import *   
d = array([2,3,4,5])   
d.nbytes
#Output: 32

Hast

Answer 3

UPDATE: Here is another, maybe more thorough recipe for estimating the size of a python object.

Here is a thread addressing a similar question

The solution proposed is to write your own... using some estimates of the known size of primitives, python's object overhead, and the sizes of built in container types.

Since the code is not that long, here is a direct copy of it:

def sizeof(obj):
    """APPROXIMATE memory taken by some Python objects in 
    the current 32-bit CPython implementation.

    Excludes the space used by items in containers; does not
    take into account overhead of memory allocation from the
    operating system, or over-allocation by lists and dicts.
    """
    T = type(obj)
    if T is int:
        kind = "fixed"
        container = False
        size = 4
    elif T is list or T is tuple:
        kind = "variable"
        container = True
        size = 4*len(obj)
    elif T is dict:
        kind = "variable"
        container = True
        size = 144
        if len(obj) > 8:
            size += 12*(len(obj)-8)
    elif T is str:
        kind = "variable"
        container = False
        size = len(obj) + 1
    else:
        raise TypeError("don't know about this kind of object")
    if kind == "fixed":
        overhead = 8
    else: # "variable"
        overhead = 12
    if container:
        garbage_collector = 8
    else:
        garbage_collector = 0
    malloc = 8 # in most cases
    size = size + overhead + garbage_collector + malloc
    # Round to nearest multiple of 8 bytes
    x = size % 8
    if x != 0:
        size += 8-x
        size = (size + 8)
    return size

Yuji

This is pretty hard coded solution. This would not work if we have a list of large dictionaries or any other data structure! – Paphos

Answer 4

I was trying to figure out how to do just that for myself. I tried several solutions on this page and other pages. I then did some search and came across https://ipython-books.github.io/44-profiling-the-memory-usage-of-your-code-with-memory_profiler/ which seems to give an alternative solution. The gist of the solution: Use %mprun in ipython.

1. First, install memory_profiler: pip install memory_profiler
2. Start ipython and load memory_profiler: %load_ext memory_profiler
3. Create a function in a physical file, say, myfunc.py (important: %mprun can only be used on functions defined in physical files). Create the object in question in a function, e.g.:

# myfunc.py
def myfunc():
    # create the object, e.g.
    a = [*range(10000)]

4. Run

from myfunc import myfunc
%mprun -T mprof -f myfunc myfunc()

which generates the file mprof. The content is also displayed:

Line #    Mem usage    Increment   Line Contents
================================================
     1     49.1 MiB     49.1 MiB   def myfunc():
     2                                 # create the object, e.g.
     3     49.4 MiB      0.3 MiB       a = [*range(10000)]

From the inrement in line 3, we know the memory used by a is 0.3 MiB.

Lets try a = [*range(100000)]:

# myfunc1.py
def myfunc1():
    # create the object, e.g.
    a = [*range(100000)]

Run

from myfunc1 import myfunc1
%mprun -T mprof1 -f myfunc1 myfunc1()

Line #    Mem usage    Increment   Line Contents
================================================
     1     49.2 MiB     49.2 MiB   def myfunc1():
     2                                 # create the object, e.g.
     3     52.3 MiB      3.0 MiB       a = [*range(100000)]

seems to be in line with our expectation.

Denesedengue