Allocating more memory than there exists using malloc

Question

This code snippet will allocate 2Gb every time it reads the letter 'u' from stdin, and will initialize all the allocated chars once it reads 'a'.

#include <iostream>
#include <stdlib.h>
#include <stdio.h>
#include <vector>
#define bytes 2147483648
using namespace std;
int main()
{
    char input [1];
    vector<char *> activate;
    while(input[0] != 'q')
    {
        gets (input);
        if(input[0] == 'u')
        {
            char *m = (char*)malloc(bytes);
            if(m == NULL) cout << "cant allocate mem" << endl;
            else cout << "ok" << endl;
            activate.push_back(m);
        }
        else if(input[0] == 'a')
        {
            for(int x = 0; x < activate.size(); x++)
            {
                char *m;
                m = activate[x];
                for(unsigned x = 0; x < bytes; x++)
                {
                    m[x] = 'a';
                }
            }
        }
    }
    return 0;
}

I am running this code on a linux virtual machine that has 3Gb of ram. While monitoring the system resource usage using the htop tool, I have realized that the malloc operation is not reflected on the resources.

For example when I input 'u' only once(i.e. allocate 2GB of heap memory), I don't see the memory usage increasing by 2GB in htop. It is only when I input 'a'(i.e. initialize), I see the memory usage increasing.

As a consequence, I am able to "malloc" more heap memory than there exists. For example, I can malloc 6GB(which is more than my ram and swap memory) and malloc would allow it(i.e. NULL is not returned by malloc). But when I try to initialize the allocated memory, I can see the memory and swap memory filling up till the process is killed.

-My questions:

1.Is this a kernel bug?

2.Can someone explain to me why this behavior is allowed?

Intenerate

BTW, your call to gets() causes buffer overflow. The solution is, well, throw it away. – Seedling

And you have undefined behavior. You can't be sure that the uninitialized input[0] is not q at start of main, you are just lucky. Compile with g++ -Wall. – Delivery

Answer 1

It is called memory overcommit. You can disable it by running as root:

 echo 2 > /proc/sys/vm/overcommit_memory

and it is not a kernel feature that I like (so I always disable it). See malloc(3) and mmap(2) and proc(5)

^{NB: echo 0 instead of echo 2 often -but not always- works also. Read the docs (in particular proc man page that I just linked to).}

Delivery

Thanks for the correction. Replaced 0 with 2. In practice 0 works usually well enough. – Delivery

Answer 2

from man malloc (online here):

By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available.

So when you just want to allocate too much, it "lies" to you, when you want to use the allocated memory, it will try to find enough memory for you and it might crash if it can't find enough memory.

Comma

Answer 3

No, this is not a kernel bug. You have discovered something known as late paging (or overcommit).

Until you write a byte to the address allocated with malloc (...) the kernel does little more than "reserve" the address range. This really depends on the implementation of your memory allocator and operating system of course, but most good ones do not incur the majority of kernel overhead until the memory is first used.

The hoard allocator is one big offender that comes to mind immediately, through extensive testing I have found it almost never takes advantage of a kernel that supports late paging. You can always mitigate the effects of late paging in any allocator if you zero-fill the entire memory range immediately after allocation.

Real-time operating systems like VxWorks will never allow this behavior because late paging introduces serious latency. Technically, all it does is put the latency off until a later indeterminate time.

For a more detailed discussion, you may be interested to see how IBM's AIX operating system handles page allocation and overcommitment.

Roxannaroxanne

Answer 4

This is a result of what Basile mentioned, over commit memory. However, the explanation kind of interesting.

Basically when you attempt to map additional memory in Linux (POSIX?), the kernel will just reserve it, and will only actually end up using it if your application accesses one of the reserved pages. This allows multiple applications to reserve more than the actual total amount of ram / swap.

This is desirable behavior on most Linux environments unless you've got a real-time OS or something where you know exactly who will need what resources, when and why.

Otherwise somebody could come along, malloc up all the ram (without actually doing anything with it) and OOM your apps.

Another example of this lazy allocation is mmap(), where you have a virtual map that the file you're mapping can fit inside - but you only have a small amount of real memory dedicated to the effort. This allows you to mmap() huge files (larger than your available RAM), and use them like normal file handles which is nifty)

-n

Mceachern

The fact that it is desirable behavior is a matter of opinion, and I strongly disagree; IMHO memory overcommitment is always bad (because it hide bugs from developers). – Delivery

It would be great if somebody could clarify if this behavior is limited to linux or is in POSIX – Comma

It is Linux specific, not POSIX. – Delivery

Thanks for clarifying! I had forgotten it was a Linuxism. I'd disagree that overcommitment is always a 'bad thing'. It's pretty easy to see how much memory you're actually using vs. how much you've allocated.. Don't throw the baby out with the bathwater. – Mceachern

We still both agree that it is a matter of opinion! And mmap of a real file with MAP_SHARED don't use overcommit (because the file serves as "swap"). – Delivery

Answer 5

Initializing / working with the memory should work:

memset(m, 0, bytes);

Also you could use calloc that not only allocates memory but also fills it with zeros for you:

char* m = (char*) calloc(1, bytes);

Statfarad

Answer 6

1.Is this a kernel bug?

No.

2.Can someone explain to me why this behavior is allowed?

There are a few reasons:

• Mitigate need to know eventual memory requirement - it's often convenient to have an application be able to an amount of memory that it considers an upper limit on the need it might actually have. For example, if it's preparing some kind of report either of an initial pass just to calculate the eventual size of the report or a realloc() of successively larger areas (with the risk of having to copy) may significantly complicate the code and hurt performance, where-as multiplying some maximum length of each entry by the number of entries could be very quick and easy. If you know virtual memory is relatively plentiful as far as your application's needs are concerned, then making a larger allocation of virtual address space is very cheap.

• Sparse data - if you have the virtual address space spare, being able to have a sparse array and use direct indexing, or allocate a hash table with generous capacity() to size() ratio, can lead to a very high performance system. Both work best (in the sense of having low overheads/waste and efficient use of memory caches) when the data element size is a multiple of the memory paging size, or failing that much larger or a small integral fraction thereof.

• Resource sharing - consider an ISP offering a "1 giga-bit per second" connection to 1000 consumers in a building - they know that if all the consumers use it simultaneously they'll get about 1 mega-bit, but rely on their real-world experience that, though people ask for 1 giga-bit and want a good fraction of it at specific times, there's inevitably some lower maximum and much lower average for concurrent usage. The same insight applied to memory allows operating systems to support more applications than they otherwise would, with reasonable average success at satisfying expectations. Much as the shared Internet connection degrades in speed as more users make simultaneous demands, paging from swap memory on disk may kick in and reduce performance. But unlike an internet connection, there's a limit to the swap memory, and if all the apps really do try to use the memory concurrently such that that limit's exceeded, some will start getting signals/interrupts/traps reporting memory exhaustion. Summarily, with this memory overcommit behaviour enabled, simply checking malloc()/new returned a non-NULL pointer is not sufficient to guarantee the physical memory is actually available, and the program may still receive a signal later as it attempts to use the memory.

Fabaceous