Total memory used by Python process?

Question

Is there a way for a Python program to determine how much memory it's currently using? I've seen discussions about memory usage for a single object, but what I need is total memory usage for the process, so that I can determine when it's necessary to start discarding cached data.

Recha

Answer 1

Here is a useful solution that works for various operating systems, including Linux, Windows, etc.:

import psutil
process = psutil.Process()
print(process.memory_info().rss)  # in bytes 

Notes:

• do pip install psutil if it is not installed yet

• handy one-liner if you quickly want to know how many MiB your process takes:

  import os, psutil; print(psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2)
  

• with Python 2.7 and psutil 5.6.3, it was process.memory_info()[0] instead (there was a change in the API later).

Wardwarde

psutil is cross platform and can return the same values as the ps command line tool: pythonhosted.org/psutil/#psutil.Process.memory_info – Ecklund

"(psutil) currently supports Linux, Windows, OSX, FreeBSD and Sun Solaris, both 32-bit and 64-bit architectures, with Python versions from 2.6 to 3.4" from Documentation – Garey

Why doesn't this number match the one in the process explorer? The number from psutil always seems to be larger by about 10%. – Tautologize

Note that psutil is not in the standard library – Succinate

@Tautologize my guess is that the process monitoring program (I'm on OS X, but likely the same on Linux and Windows) is sampling the memory usage over time and missing the absolute peak – Duhl

If using Ubuntu, you can install psutil with apt: sudo apt install python3-psutil – Agrippina

For recent versions of psutil, psutil.Process() is equivalent to psutil.Process(os.getpid()). That's one less thing you need to remember to type. – Exquisite

Why do you use rss? In psutil document: rss | resident set size, vms | total program size. So I think last line should be print(process.memory_info().vms) – Synecology

Totally agree to @HalilİbrahimOymacı: RSS is physical memory. That's the wrong metric in >90% of the cases. A lot of effects cannot be observed. And: A high RSS is good, because you actually make use of RAM. That's better than use of hard disk, because it's faster. – Allare

This answer explains difference between resident set size and virtual memory size very well. – Ferreous

To print memory usage in human readable form using the humanize package: humanize.naturalsize(psutil.Process(os.getpid()).memory_info().rss). Example 2.9 GB – Liris

Answer 2

For Unix based systems (Linux, Mac OS X, Solaris), you can use the getrusage() function from the standard library module resource. The resulting object has the attribute ru_maxrss, which gives the peak memory usage for the calling process:

>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
2656  # peak memory usage (kilobytes on Linux, bytes on OS X)

The Python docs don't make note of the units. Refer to your specific system's man getrusage.2 page to check the unit for the value. On Ubuntu 18.04, the unit is noted as kilobytes. On Mac OS X, it's bytes.

The getrusage() function can also be given resource.RUSAGE_CHILDREN to get the usage for child processes, and (on some systems) resource.RUSAGE_BOTH for total (self and child) process usage.

If you care only about Linux, you can alternatively read the /proc/self/status or /proc/self/statm file as described in other answers for this question and this one too.

Autoerotism

Okay, will do. I wasn't sure if SO had a process for merging questions or what. The duplicate post was partly to show people there was a standard library solution on both questions... and partly for the rep. ;) Should I delete this answer? – Autoerotism

Oh, it appears I can't flag duplicates with my current rep - the "flag" link under questions doesn't even appear for me. – Autoerotism

Now you've got more than enough rep to flag :) – Sauna

Note: On linux ru_maxrss returns a value indicating the number of memory pages used. ru_idrss returns private memory use in kilobytes. – Tedi

tried to use ru_maxrss and ru_idrss, in both cases got incorrect results for heavy processes. When memory consumption of my process exceeds 4GB, the counter resets to zero. I expect that this is related to 64-bit architecture, will try other solutions. OS X 10.8 64bit. – Buttons

The ru_maxrss reports the wrong number (doesn't correspond to ps aux <PID> before and after wasting some memory) on Mac OS, and all of the other .ru_*rss properties return None. – Burroughs

Mac OS definitely returns the RSS in bytes, Linux returns it in kilobytes. – Burroughs

The units are NOT in kilobytes. It is platform dependent, so you have to use resource.getpagesize() to find out. The given Python docs (docs.python.org/2/library/resource.html#resource-usage) is actually very clear about it. It is 4096 in my box. – Legalize

(And on OSX, even if resource.getpagesize() reports 4096, the units of ru_maxrss are still single bytes.) – Winnick

@BenLin Those Python docs are clearly wrong, or there is a bug on the Mac version. The unit used by getrusage and the value returned by getpagesize are definitely different. – Sprinkler

The question asked for current usage. Note that this is maximum usage. (Still a useful answer, just warning people who mistakenly copy-paste it.) – Dhiren

That's weird, both the Linux man pages <linux.die.net/man/2/getrusage> and the Mac (well, iPhoneOS) man pages <developer.apple.com/library/archive/documentation/System/…> say ru_maxrss is in kilobytes. No mention of bytes or pages. – Translunar

According to linux.die.net/man/2/getrusage a lot of the fields are not set in linux. (eg. the fields related to the current memory usage). – Corbett

Answer 3

On Windows, you can use WMI (home page, cheeseshop):

def memory():
    import os
    from wmi import WMI
    w = WMI('.')
    result = w.query("SELECT WorkingSet FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=%d" % os.getpid())
    return int(result[0].WorkingSet)

On Linux (from python cookbook http://code.activestate.com/recipes/286222/:

import os
_proc_status = '/proc/%d/status' % os.getpid()

_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0, 'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
    '''Private.'''
    global _proc_status, _scale
     # get pseudo file  /proc/<pid>/status
    try:
        t = open(_proc_status)
        v = t.read()
        t.close()
    except:
        return 0.0  # non-Linux?
     # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'
    i = v.index(VmKey)
    v = v[i:].split(None, 3)  # whitespace
    if len(v) < 3:
        return 0.0  # invalid format?
     # convert Vm value to bytes
    return float(v[1]) * _scale[v[2]]

def memory(since=0.0):
    '''Return memory usage in bytes.'''
    return _VmB('VmSize:') - since

def resident(since=0.0):
    '''Return resident memory usage in bytes.'''
    return _VmB('VmRSS:') - since

def stacksize(since=0.0):
    '''Return stack size in bytes.'''
    return _VmB('VmStk:') - since

Metalware

The Windows code doesn't work for me. This change does: return int(result[0].WorkingSet) – Kort

This Windows code doesn't work for me on Windows 7 x64, even after John Fouhy's comment modification. – Wardwarde

I have this error: return [ wmi_object (obj, instance_of, fields) for obj in self._raw_query(wql) ] File "C:\Python27\lib\site-packages\win32com\client\util.py", line 84, in next return _get_good_object_(self._iter.next(), resultCLSID = self.resultCLSID) pywintypes.com_error: (-2147217385, 'OLE error 0x80041017', None, None) if anyone can help me? I have win 8 x64 but python on x32 – Astrodome

Note: I updated the windows example following John Fouhy's suggestion after inspecting the (latest) source of the wmi module. See also (1), (2). – Husha

@jedwards: WorkingSet is a very bad number in >90% of the cases. It's just that part of memory which is currently in RAM – Allare

Does the Window's code return amount of RAM in use in bytes/kilobytes/etc? – Misgiving

Bytes. See wutils.com/wmi/root/cimv2/win32_perfrawdata_perfproc_process/… – Metalware

Answer 4

On unix, you can use the ps tool to monitor it:

$ ps u -p 1347 | awk '{sum=sum+$6}; END {print sum/1024}'

where 1347 is some process id. Also, the result is in MB.

Spurt

Answer 5

Current memory usage of the current process on Linux, for Python 2, Python 3, and pypy, without any imports:

def getCurrentMemoryUsage():
    ''' Memory usage in kB '''

    with open('/proc/self/status') as f:
        memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]

    return int(memusage.strip())

It reads the status file of the current process, takes everything after VmRSS:, then takes everything before the first newline (isolating the value of VmRSS), and finally cuts off the last 3 bytes which are a space and the unit (kB).
To return, it strips any whitespace and returns it as a number.

Tested on Linux 4.4 and 4.9, but even an early Linux version should work: looking in man proc and searching for the info on the /proc/$PID/status file, it mentions minimum versions for some fields (like Linux 2.6.10 for "VmPTE"), but the "VmRSS" field (which I use here) has no such mention. Therefore I assume it has been in there since an early version.

Dhiren

Answer 6

Below is my function decorator which allows to track how much memory this process consumed before the function call, how much memory it uses after the function call, and how long the function is executed.

import time
import os
import psutil


def elapsed_since(start):
    return time.strftime("%H:%M:%S", time.gmtime(time.time() - start))


def get_process_memory():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss


def track(func):
    def wrapper(*args, **kwargs):
        mem_before = get_process_memory()
        start = time.time()
        result = func(*args, **kwargs)
        elapsed_time = elapsed_since(start)
        mem_after = get_process_memory()
        print("{}: memory before: {:,}, after: {:,}, consumed: {:,}; exec time: {}".format(
            func.__name__,
            mem_before, mem_after, mem_after - mem_before,
            elapsed_time))
        return result
    return wrapper

So, when you have some function decorated with it

from utils import track

@track
def list_create(n):
    print("inside list create")
    return [1] * n

You will be able to see this output:

inside list create
list_create: memory before: 45,928,448, after: 46,211,072, consumed: 282,624; exec time: 00:00:00

Profit

Answer 7

Even easier to use than /proc/self/status: /proc/self/statm. It's just a space delimited list of several statistics. I haven't been able to tell if both files are always present.

/proc/[pid]/statm

Provides information about memory usage, measured in pages. The columns are:

• size (1) total program size (same as VmSize in /proc/[pid]/status)
• resident (2) resident set size (same as VmRSS in /proc/[pid]/status)
• shared (3) number of resident shared pages (i.e., backed by a file) (same as RssFile+RssShmem in /proc/[pid]/status)
• text (4) text (code)
• lib (5) library (unused since Linux 2.6; always 0)
• data (6) data + stack
• dt (7) dirty pages (unused since Linux 2.6; always 0)

Here's a simple example:

from pathlib import Path
from resource import getpagesize

PAGESIZE = getpagesize()
PATH = Path('/proc/self/statm')


def get_resident_set_size() -> int:
    """Return the current resident set size in bytes."""
    # statm columns are: size resident shared text lib data dt
    statm = PATH.read_text()
    fields = statm.split()
    return int(fields[1]) * PAGESIZE


data = []
start_memory = get_resident_set_size()
for _ in range(10):
    data.append('X' * 100000)
    print(get_resident_set_size() - start_memory)

That produces a list that looks something like this:

0
0
368640
368640
368640
638976
638976
909312
909312
909312

You can see that it jumps by about 300,000 bytes after roughly 3 allocations of 100,000 bytes.

Docker

For those in need of higher accuracy, documentation suggests using /proc/[pid]/smaps, but that's a bit more complicated. – Zwickau

Answer 8

For Python 3.6 and psutil 5.4.5 it is easier to use memory_percent() function listed here.

import os
import psutil
process = psutil.Process(os.getpid())
print(process.memory_percent())

Gastroenterology

this requires lib psutil – Lidalidah

Answer 9

I like it, thank you for @bayer. I get a specific process count tool, now.

# Megabyte.
$ ps aux | grep python | awk '{sum=sum+$6}; END {print sum/1024 " MB"}'
87.9492 MB

# Byte.
$ ps aux | grep python | awk '{sum=sum+$6}; END {print sum " KB"}'
90064 KB

Attach my process list.

$ ps aux  | grep python
root       943  0.0  0.1  53252  9524 ?        Ss   Aug19  52:01 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
root       950  0.6  0.4 299680 34220 ?        Sl   Aug19 568:52 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
root      3803  0.2  0.4 315692 36576 ?        S    12:43   0:54 /usr/bin/python /usr/local/bin/beaver -c /etc/beaver/beaver.conf -l /var/log/beaver.log -P /var/run/beaver.pid
jonny    23325  0.0  0.1  47460  9076 pts/0    S+   17:40   0:00 python
jonny    24651  0.0  0.0  13076   924 pts/4    S+   18:06   0:00 grep python

Reference

• memory - Linux: find out what process is using all the RAM? - Super User
• Total memory used by Python process? - Stack Overflow
• linux - ps aux output meaning - Super User

Zak

just an optimisation of code to avoid multi pipe ps aux | awk '/python/{sum+=$6}; END {print sum/1024 " MB"}' – Cymogene

Answer 10

import os, win32api, win32con, win32process
han = win32api.OpenProcess(win32con.PROCESS_QUERY_INFORMATION|win32con.PROCESS_VM_READ, 0, os.getpid())
process_memory = int(win32process.GetProcessMemoryInfo(han)['WorkingSetSize'])

Interlunation

This could be improved with some explanation of what it does and how it works. – Aranyaka

Based on the large number returned (8 digits) and how I'm not doing much of anything, I'm guessing this has to be bytes? So it's around 28.5 MB for a rather idle interactive instance. (Wow... I didn't even realize the above comment was mine from 4 years ago... that's weird.) – Aranyaka

Answer 11

For Unix systems command time (/usr/bin/time) gives you that info if you pass -v. See Maximum resident set size below, which is the maximum (peak) real (not virtual) memory that was used during program execution:

$ /usr/bin/time -v ls /

    Command being timed: "ls /"
    User time (seconds): 0.00
    System time (seconds): 0.01
    Percent of CPU this job got: 250%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 0
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 315
    Voluntary context switches: 2
    Involuntary context switches: 0
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Nellanellda

Note that this may fail if you just try to use time instead of /usr/bin/time. See: askubuntu.com/questions/434289/… – Hottempered

Answer 12

Using sh and os to get into python bayer's answer.

float(sh.awk(sh.ps('u','-p',os.getpid()),'{sum=sum+$6}; END {print sum/1024}'))

Answer is in megabytes.

Aftershaft

Should be noted that `sh' isn't a stdlib module. It's installable with pip, though. – Govan