Force free() to return malloc memory back to OS

Question

Seems like even after I free all the memory for a Linux process that was allocated by malloc(), memory is still reserved for the process and not returned to the OS.

Running valgrind massif tool by default reveals no leakages.

Running valgrind with --pages-as-heap=yes reveals this:

->13.77% (7,655,424B) 0x35FEEEB069: brk (brk.c:31)

->13.77% (7,655,424B) 0x35FEEEB113: sbrk (sbrk.c:53)

->13.77% (7,655,424B) 0x35FEE82717: __default_morecore (morecore.c:48)

->13.77% (7,655,424B) 0x35FEE7DCCB: _int_malloc (malloc.c:2455)

->13.77% (7,655,424B) 0x35FEE7F4F1: malloc (malloc.c:2862)

so even though memory was already freed by free(), it seems that malloc called brk/sbrk and did not return this to the OS.

how can I force free() to call sbrk() immediately and return all memory back to the OS ?

I am running on a very low end platform which every MB counts.

Thanks in advance.

Nowak

malloc - memory allocate. it only allocates, it doesn't "free" memory for re-use, which is why there's free(). – Dorsum

hmm, seems like this might be the solution: #2215759 – Nowak

This is not a problem with your program or the malloc/free calls, it's the operating system that keeps the previously allocated memory-pages mapped to your process. If the OS needs those free pages it will take them when needed. I'm to lazy to find a duplicate, but there are many duplicates of this question. – Wiburg

@JoachimPileborg actually this is not working apparently. i got a crash from Linux kernel saying there is not enough memory. when i called my process with fork and it was killed, sbrk was released and i was able to continue. so the OS did not call sbrk to release the previously freed memory – Nowak

How much memory do you allocate? Did you free that memory before forking? When you try to fork, how much free memory was available? – Wiburg

@JoachimPileborg see this man7.org/linux/man-pages/man3/mallopt.3.html M_TRIM_THRESHOLD this answer the question - malloc waits for enough contigious free memory to call sbrk and return the memory back to the OS – Nowak

The title in this question is confusing, it made me think that the OP didn't now about free(). – Sequential

You should have a look to #2215759 – Laval

Answer 1

With glibc malloc try to call malloc_trim function. It is not well documented and there were changes inside it at around 2007 (glibc 2.9) - https://stackoverflow.com/a/42281428.

Since 2007 this function will: Iterate over all malloc memory arenas (used in multithreaded applications) doing trim and fastbin consolidation; and release all aligned (4KB) pages fully freed.

https://sourceware.org/git/?p=glibc.git;a=commit;f=malloc/malloc.c;h=68631c8eb92ff38d9da1ae34f6aa048539b199cc

Ulrich Drepper Sun, 16 Dec 2007 22:53:08 +0000 (22:53 +0000)

• malloc/malloc.c (public_mTRIm): Iterate over all arenas and call mTRIm for all of them.

(mTRIm): Additionally iterate over all free blocks and use madvise to free memory for all those blocks which contain at least one memory page.

https://sourceware.org/git/?p=glibc.git;a=blobdiff;f=malloc/malloc.c;h=c54c203cbf1f024e72493546221305b4fd5729b7;hp=1e716089a2b976d120c304ad75dd95c63737ad75;hb=68631c8eb92ff38d9da1ae34f6aa048539b199cc;hpb=52386be756e113f20502f181d780aecc38cbb66a

+  malloc_consolidate (av);
...
+  for (int i = 1; i < NBINS; ++i)
...
+        for (mchunkptr p = last (bin); p != bin; p = p->bk)
+         {
...
+               /* See whether the chunk contains at least one unused page.  */
+               char *paligned_mem = (char *) (((uintptr_t) p
+                                               + sizeof (struct malloc_chunk)
+                                               + psm1) & ~psm1);
...
+               /* This is the size we could potentially free.  */
+               size -= paligned_mem - (char *) p;
+
+               if (size > psm1)
+                 {
...
+                   madvise (paligned_mem, size & ~psm1, MADV_DONTNEED);

So, calling malloc_trim will release almost all freed memory back to the OS. Only pages containing still not freed data will be kept; OS may unmap or not unmap physical page when madvised with MADV_DONTNEED and linux usually does unmap. madvised pages are still count to VSIZE (total virtual memory size of the process), but usually help to reduce RSS (amount of physical memory used by process).

Alternatively, you can try to switch into alternative malloc library: tcmalloc (gperftools / google-perftools) or jemalloc (facebook), both of them have aggressive rules of returning freed memory back to OS (with madvise MADV_DONTNEED or even MADV_FREE).

Medwin

Answer 2

The only reliable and portable way to have the OS reclaim memory is to exit the process and restart it again, restoring any state you need to continue.

Of course, writing your own malloc/free implementation using brk/sbrk according to your needs is the other option.

Seanseana