Fail to malloc big block memory after many malloc/free small blocks memory

Question

Here is the code.

First I try to malloc and free a big block memory, then I malloc many small blocks memory till it run out of memory, and I free ALL those small blocks.

After that, I try to malloc a big block memory.

#include <stdio.h>
#include <stdlib.h>
int main (int argc, char **argv)
{
    static const int K = 1024;
    static const int M = 1024 * K;
    static const int G = 1024 * M;

    static const int BIG_MALLOC_SIZE = 1 * G;
    static const int SMALL_MALLOC_SIZE = 3 * K;
    static const int SMALL_MALLOC_TIMES = 1 * M;

    void **small_malloc = (void **)malloc(SMALL_MALLOC_TIMES * sizeof(void *));

    void *big_malloc = malloc(BIG_MALLOC_SIZE);
    printf("big malloc first time %s\n", (big_malloc == NULL)? "failed" : "succeeded");
    free(big_malloc);

    for (int i = 0; i != SMALL_MALLOC_TIMES; ++i)
    {
        small_malloc[i] = malloc(SMALL_MALLOC_SIZE);
        if (small_malloc[i] == NULL)
        {
            printf("small malloc failed at %d\n", i);
            break;
        }
    }
    for (int i = 0; i != SMALL_MALLOC_TIMES && small_malloc[i] != NULL; ++i)
    {
        free(small_malloc[i]);
    }

    big_malloc = malloc(BIG_MALLOC_SIZE);
    printf("big malloc second time %s\n", (big_malloc == NULL)? "failed" : "succeeded");
    free(big_malloc);

    return 0;
}

Here is the result:

big malloc first time succeeded
small malloc failed at 684912
big malloc second time failed

It looks like there are memory fragments.

I know memory fragmentation happens when there are many small empty space in memory but there is no big enough empty space for big size malloc.

But I've already free EVERYTHING I malloc, the memory should be empty.

Why I can't malloc big block at the second time?

I use Visual Studio 2010 on Windows 7, I build 32-bits program.

Neurotomy

Perhaps the memory is still fragmented. Have you tried to monitor what happens to memory while your program is running? – Blatherskite

Note: You are freeing uninitialized pointers (after a small malloc fail) – Seamaid

@DieterLücking I thought so too, but the free()-loop ends when the first NULL is encountered. – Carlson

Although I'm not sure it's enough to fix this code, when VC++ has problems due to heap fragmentation, you can sometimes help by calling _heapmin(). – Characterize

@DieterLücking When smalloc malloc fail, malloc return NULL. In the second for loop I check small_malloc[i] != NULL to guard the uninitialized pointers. – Neurotomy

Obviously, VC++ fails to re-combine adjacent free memory blocks. With GCC on Linux 32 bit, it works. – Marchand

The problem is not VC++ but libc's heap manager. By default, windows' heap manager is really conservative, but there's an API to request the use of the new heap manager – Unconventional

Any chance it is due to using of the debug version of C runtime library? – Dwyer

@Dwyer I double check that it's the release version. – Neurotomy

@Neurotomy use size_t not int for storing your malloc values, int is not capable of holding the size of allocable memory on the system, and its also signed so you risk overflow. – Henninger

Answer 1

The answer, sadly, is still fragmentation.

Your initial large allocation ends up tracked by one allocation block; however when you start allocating large numbers of 3k blocks of memory your heap gets sliced into chunks.

Even when you free the memory, small pieces of the block remain allocated within the process's address space. You can use a tool like Sysinternals VMMap to see these allocations visually.

It looks like 16M blocks are used by the allocator, and once these blocks are freed up they never get returned to the free pool (i.e. the blocks remain allocated).

As a result you don't have enough contiguous memory to allocate the 1GB block the second time.

Benjie

Answer 2

Even I know just a little about this, I found the following thread Why does malloc not work sometimes? which covers the similar topic as yours. It contains the following links: http://www.eskimo.com/~scs/cclass/int/sx7.html (Pointer Allocation Strategies) http://www.gidforums.com/t-9340.html (reasons why malloc fails? )

Karwan

just a comment: I tried your code under Win 7 32 bit/4GB built in CVI 2013 and there is no problem as you described, so it is probably VC2010 (or itssub-part) specific. – Karwan

Answer 3

The issue is likely that even if you free every allocation, malloc does not return all the memory to the operating system.

When your program requested the numerous smaller allocations, malloc had to increase the size of the "arena" from which it allocates memory.

There is no guarantee that if you free all the memory, the arena will shrink to the original size. It's possible that the arena is still there, and all the blocks have been put into a free list (perhaps coalesced into larger blocks).

The presence of this lingering arena in your address space may be making it impossible to satisfy the large allocation request.

Umpteen