Limit RAM usage to python program

Question

I'm trying to limit the RAM usage from a Python program to half so it doesn't totally freezes when all the RAM is used, for this I'm using the following code which is not working and my laptop is still freezing:

import sys
import resource

def memory_limit():
    rsrc = resource.RLIMIT_DATA
    soft, hard = resource.getrlimit(rsrc)
    soft /= 2
    resource.setrlimit(rsrc, (soft, hard))

if __name__ == '__main__':
    memory_limit() # Limitates maximun memory usage to half
    try:
        main()
    except MemoryError:
        sys.stderr.write('MAXIMUM MEMORY EXCEEDED')
        sys.exit(-1)

I'm using other functions which I call from the main function.

What am I doing wrong?

Thanks in advance.

PD: I already searched about this and found the code I've put but it's still not working...

Guise

Perhaps you want ulimit or prlimit outside the Python script. Or set up a memory limited cgroup and run the script there. I'm not sure trying to self-limit is the best idea - what happens if the code that tries to check or enforce the limit needs to allocate memory in the process? – Nadaha

Did you try soft /= 100, or soft //= 2 ? – These

if I use that it says exits too soon @These – Guise

why is all your RAM being used? Are you loading a lot of data into memory? Have you tried using generators? – Electrodeposit

@Electrodeposit it's for a university's homework, we must generate a tree using a DFS straetgy which gets into an infinite loop because of the task statement. – Guise

Answer 1

I've done some research and found a function to get the memory from Linux systems here: Determine free RAM in Python and I modified it a bit to set the memory hard limit to half of the free memory available.

import resource
import sys

def memory_limit_half():
    """Limit max memory usage to half."""
    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    # Convert KiB to bytes, and divide in two to half
    resource.setrlimit(resource.RLIMIT_AS, (int(get_memory() * 1024 / 2), hard))

def get_memory():
    with open('/proc/meminfo', 'r') as mem:
        free_memory = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                free_memory += int(sline[1])
    return free_memory  # KiB

if __name__ == '__main__':
    memory_limit_half()
    try:
        main()
    except MemoryError:
        sys.stderr.write('\n\nERROR: Memory Exception\n')
        sys.exit(1)

Guise

Hi Ulises CT, is there any reason for adding up MemFree, Buffers, and Cached, instead of just taking the value of MemAvailable? – Antony

Answer 2

I modify the answer of @Ulises CT. Because I think to change too much original function is not so good, so I turn it to a decorator. I hope it helps.

import resource
import platform
import sys

def memory_limit(percentage: float):
    """
    只在linux操作系统起作用
    """
    if platform.system() != "Linux":
        print('Only works on linux!')
        return
    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    resource.setrlimit(resource.RLIMIT_AS, (get_memory() * 1024 * percentage, hard))

def get_memory():
    with open('/proc/meminfo', 'r') as mem:
        free_memory = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                free_memory += int(sline[1])
    return free_memory

def memory(percentage=0.8):
    def decorator(function):
        def wrapper(*args, **kwargs):
            memory_limit(percentage)
            try:
                return function(*args, **kwargs)
            except MemoryError:
                mem = get_memory() / 1024 /1024
                print('Remain: %.2f GB' % mem)
                sys.stderr.write('\n\nERROR: Memory Exception\n')
                sys.exit(1)
        return wrapper
    return decorator

@memory(percentage=0.8)
def main():
    print('My memory is limited to 80%.')

Appoint

in the memory decorator you should return function(*args, **kwargs) – Sabol

Is there any other way to make this decorator to work with all OS? – Swinton

Answer 3

due to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773

it is better to use :

if str(sline[0]) == 'MemAvailable:':
            free_memory = int(sline[1])
            break

the answer code provided me with 8 GB of available mem on a machine with 1TB of RAM

Ophir

Answer 4

A simpler solution is:

import psutil
import resource

# Calculate the maximum memory limit (80% of available memory)
virtual_memory = psutil.virtual_memory()
available_memory = virtual_memory.available
memory_limit = int(available_memory * 0.8)

# Set the memory limit
resource.setrlimit(resource.RLIMIT_AS, (memory_limit, memory_limit))

Mislike