Resident Set Size (RSS) limit has no effect

Question

The following problem occurs on a machine running Ubuntu 10.04 with the 2.6.32-22-generic kernel: Setting a limit for the Resident Set Size (RSS) of a process does not seem to have any effect. I currently set the limit in Python with the following code:

import resource
# (100, 100) is the (soft, hard) limit. ~100kb.
resource.setrlimit(resource.RLIMIT_RSS, (100, 100))
memory_sink = ['a']*10000000   # this should fail

The list, memory_sink, succeeds every time. When I check RSS usage with top, I can easily get the process to use 1gb of RAM, which means that the limit is not working. Do RSS limits not work with this kernel or distro? If it helps, resource.RLIMIT_NPROC (user process limit) does work.

Hovercraft

Answer 1

Form the getrlimit manpage:

RLIMIT_RSS
Specifies the limit (in pages) of  the  process's  resident  set
(the  number of virtual pages resident in RAM).  This limit only
has effect in Linux 2.4.x, x < 30, and there only affects  calls
to madvise(2) specifying MADV_WILLNEED.

It seems this is just not supported on Linux kernel 2.6.

Kershaw

I had googled the man page countless times, but missed the ones that had this provision. A simple "man getrlimit" on my machine would have sufficed. Thanks! – Hovercraft

Answer 2

You can accomplish this using cgroups. The long version is on my blog, but the short version (tested on Ubuntu 11.04) is:

• Install the cgroup-bin package.

• Edit /etc/cgconfig.config and create a group with limited memory. For instance, I added:

  group limited {
    memory {
      memory.limit_in_bytes = 50M;
    }
  }
  

• Run

  $ sudo restart cgconfig
  $ sudo chown -R jlebar /sys/fs/cgroup/memory/limited
  $ cgexec -g memory:limited your/program
  

I observed my process with an RSS of 93M when I asked it to use only 50M, but that wasn't a problem for me, since my goal was just to get the program to page.

cgclassify lets you attach restrictions to a running process too. Note for RSS this only applies to memory allocated after the restriction comes into effect.

Tyrosine

On Ubuntu 11.10 the config file location changed to: /etc/cgconfig.conf – Splenomegaly

On Ubuntu 10.04 I just get the error message restart: Unknown job: cgconfig when trying to run the sudo restart cgconfig command. :( – Detergency

I do not have root. Can I still use this? – Meantime

For newer Ubuntus (I tried on 16.04): askubuntu.com/a/848022/65575 (cgconfigparser -l /etc/cgconfig.config) – Rieger

Answer 3

A related limit - virtual memory or address space (RLIMIT_AS) - does work. This allows limiting the python process and subprocesses memory without external tools.

>>> size = 50*1024*1024 # In bytes
>>> resource.setrlimit(resource.RLIMIT_AS, (size, resource.RLIM_INFINITY))
>>> a = 'a' * size
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError

From the man page:

RLIMIT_AS. The maximum size of the process's virtual memory (address space) in bytes.

Here is a good explanation of the difference between the Resident Set and the VM size - What is RSS and VSZ in Linux memory management.

Despondency

When I try to use RLIMIT_AS, I get a MemoryError even when allocating (much) less than the limit. See this question. – Meantime

@Meantime This might come from the fact that RLIMIT_AS "includes all memory that the process can access, including memory that is swapped out, memory that is allocated, but not used, and memory that is from shared libraries." https://mcmap.net/q/25489/-what-is-rss-and-vsz-in-linux-memory-management – Schmit

Answer 4

I created a script to limit memory usage using cgroups and cgroup manager, useable for ad-hoc commands and not needing root privileges. See https://unix.stackexchange.com/questions/134414/how-to-limit-the-total-resources-memory-of-a-process-and-its-children/174894#174894

Datnow