glibc application holding onto unused memory until just before exit

Question

I have a C++ application (gcc 4.9.1, glibc 2.17) running on Linux (Centos 7). It uses various third-party libraries, notably Boost 1.61. As the application runs, I can watch its memory usage increasing steadily via htop's VIRT and RES columns, or the ps command, etc. If I let it go long enough, it will use enormous amounts of that memory and swamp the box.

Sounds like a leak, but it passes valgrind with only a few bytes leaked, all in places I'd expect. Debug print messages indicate program flow is as expected.

Digging further via the debugger, I find that most of this memory is still in use when __run_exit_handlers is invoked at the end of main. I can step through various calls to free as it works through the global destructor chain. After it finishes those, I observe only a minimal downward change in the apparent memory usage. Then, finally it calls _exit(), and only then is the memory restored to the operating system, all at once.

Can anyone offer me additional tips on how to proceed debugging this? Why won't my program give that memory back?

Shanna

How do you know it's a memory leak, and not that your application is just memory intensive? – Phew

I don't think it is a leak. Note that this memory is still "in use" after my application's code is done running, and according to valgrind at least, has released all the memory it asked for. – Shanna

@Ivan I'm not allocating many objects at all, but one of the 3p libraries I am using could be. I'm going to have to dig deeper there. – Shanna

Another question about this. – Tidewater

Note that while it isn't technically a leak, it is possible to continue using more and more memory. Maybe the program forgets to free it. Maybe memory fragmentation means lots of small free pieces that cannot be reused. – Bobsleigh

Why do you expect your program to "give memory back"? It is quite possible that there's no memory leak in your code, it is just that C++ run-time library keep holding on to memory requested from the system (since giving it back to OS and then re-requesting it is a relatively expensive operation). Why do you consider this to be a problem? I can see this being an issue on RAM-only systems (like Android), but this is a total non-issue on swap-enabled systems like desktop Linux. – Lactary

@AnT I expect it to give back the memory it's no longer using. There is no leak in my application code. The problem is that glibc is not returning a resource to the operating system. – Shanna

@John S: Well, again, why do you see it as a "problem"? Why are you concerned about it? – Lactary

Answer 1

Everything here is based on GNU libc implementation of malloc running on Linux.

Test program below does not seem to give away any memory to the system after freeing memory (strace does not show sbrk calls that return memory back to the kernel):

int main()
{
    static const int N = 5000000;
    static void *arr[N];

    for (int i = 0; i < N; i++)
        arr[i] = std::malloc(1024);

    // reverse to simplify allocators job
    for (int i = N - 1; i >= 0; i--)
        std::free(arr[i]);
}

Looks like glibc does not give away memory back at all. According to mallopt(3) man page, parameter M_TRIM_THRESHOLD is responsible for giving away memory. By default it is 128kb, while test program allocates and frees 5 GB of memory. Looks like some other details of malloc implementation do not let it free memory.

At the moment I can recommend following solutions:

• If you can, try calling malloc_trim once in a while or after freeing a lot of memory. This should force trimming and should give memory back to OS using MADV_DONTNEED.
• Avoid allocating large number of small objects using malloc or operator new, instead allocate them from a memory pool of a size greater than M_MMAP_THRESHOLD. Try destroing that pool afterwards if program logic allows this. Memory chunks of size greater than M_MMAP_THRESHOLD are immediately released back to OS.
• Same as previous one, but should be faster: allocate memory pools for small objects using mmap and release memory back to OS using madvise and MADV_DONTNEED/MADV_FREE.
• Try using another allocator that might take advantage of MADV_FREE to return memory back to the system (jemalloc?).

---

I have found this old (2006) ticket on glibc's bugzilla. It says there that free never returns memory back to the kernel, unless malloc_trim is called.

Newer versions of free seem to have code that executes internal systrim function that should trim top of the arena, but I wasn't able to make it work.

Tidewater

This is helpful. Periodic calls to malloc_trim largely reduced the amount of RES memory in use by the application, but not VIRT. I'm not allocating many objects in my own code, so one of the external libraries I'm using must be. I'm going to have to profile them, see if I can modify them to use a pool allocator in just the right spot. – Shanna

@JohnS Virtual memory should not be such a big problem. The problem with malloc_trim is that might not free any virtual memory at all, instead it will use madvise with MADV_DONTNEED on free regions, replacing unused pages zero page. So OS gets its memory back, but changes in address space take time, reducing performance. MADV_FREE does similar thing, but should be faster since it remaps pages only under memory pressure. – Tidewater

The trim threshold is for single allocations. If an application uses 5 GB of 128 byte allocations, that doesn't get trimmed on free. – Bobsleigh

@ZanLynx That means since M_MMAP_THRESHOLD is 128kb by default as well as M_TRIM_THRESHOLD, memory is never released with default parameters for small allocations. – Tidewater

@Ivan: That's right. The assumption is that future allocations will reuse those free memory bits. – Bobsleigh

Answer 2

You can profile your memory allocation using valgrind --tool=massif ./executable

Check out the documentation at http://valgrind.org/docs/manual/ms-manual.html

Then once you have profiling data you can apply memory pools and other techniques. Since you already use Boost you can find several such tools in Boost.

Bobsleigh