Measure (max) memory usage with IPython—like timeit but memit

Question

I have a simple task: in addition to measuring the time it takes to execute a chunk of code in Python, I need to measure the amount of memory a given chunk of code needs.

IPython has a nice utility called timeit which works like this:

In [10]: timeit 3 + 3
10000000 loops, best of 3: 24 ns per loop

What I'm looking for is something like this:

In [10]: memit 3 + 3
10000000 loops, best of 3: 303 bytes per loop

I'm aware that this probably does not come built in with IPython—but I like the timeit-memit analogy.

Costrel

I'll take a look, thanks; the solution there looks quite simple. And good point about "best"—but timeit also gives you the best, and you might say you're really (sometimes) interested in the worst time. Re. IPython magic: I've found blog.vene.ro/2012/06/30/… – Costrel

The question that this question has been marked as duplicate of is about memory profilers in Python in general—this one here asks about IPython; so I don't see how it's really a duplicate... The other question has no mentions of timeit, memit or even "ipython" whatsoever. – Costrel

Answer 1

In fact, it already exists, as part of the pragmatically named memory_profiler package:

In [2]: %memit np.zeros(1e7)
maximum of 3: 76.402344 MB per loop

More info at https://github.com/pythonprofilers/memory_profiler#ipython-integration

Edit: To use this, you first need to load it as an IPython extension:

%load_ext memory_profiler

To make IPython always load the memory_profiler extension upon startup, add it to the c.InteractiveShellApp.extensions list in your profile's ipython_config.py:

$ grep -C2 c.InteractiveShellApp.extensions ~/.ipython/profile_default/ipython_config.py
 # A list of dotted module names of IPython extensions to load.
 #
 c.InteractiveShellApp.extensions = [
   'autoreload',
   'memory_profiler',

Richma

having done pip install memory_profiler, In [2]: %memit 3 gives me ERROR: Line magic function %memit` not found`. – Costrel

There's an IPython specific command to load an extension. I've added it to the answer. – Richma

I also needed to do pip install memory_profiler prior to %load_ext memory_profiler working in my Jupyter Notebook. – Unreliable

If the config file doesn’t already exist, run ipython profile create in a terminal. – Blackberry

It's important to note that I believe %%memit only measures the memory difference (increase) from before the cell contents was run, and afterwards. Therefore it does not actually measure the peak memory usage during the execution of the cell contents – Naarah