Any way to reserve but not commit memory in linux?

Question

Windows has VirtualAlloc, which allows you to reserve a contiguous region of address space, but not actually use any physical memory. Later when you want to use it (or part of it) you call VirtualAlloc again to commit the region of previously reserved pages.

This is actually really useful, but I want to eventually port my application to linux - so I don't want to use it if I can't port it later. Does linux have a way to do this?

EDIT - Use Case

I'm thinking of allocating 4 GB or some such of virtual address space, but only committing it 64K at a time. This would give me a zero-copy way to grow an array up to 4 GB. Which is important, because the typical double the array size and copy introduces seemingly random unacceptable latency for very large arrays.

Alburnum

What use case does that hold for you? Why is it important to separate the difference between asking if you can allocate the space (reserving it) and actually using the space in memory? – Eldwon

it looks like regular alloc should work fine. In case memory is not used it'll be swapped out, and, when u start to use it, it'll be brought back to mmeory – Distemper

@xyld: One deducts the block from virtual address space, the other deducts it from virtual memory (pagefile). – Homey

By the way, a fast way to solve this problem without copying is mremap (although it only starts to win out over the TLB invalidate and other costs at large buffer sizes (>65K)) – Alburnum

Reserving but not committing memory probably does not do as much as you think for reducing "random unacceptable latency." It probably creates it. You are going to introduce page faults at some point in the future and servicing those creates even more latency, which is why RTOSes usually steer clear of this. – Lagerkvist

Answer 1

mmap a special file, like /dev/zero (or use MAP_ANONYMOUS) as PROT_NONE, later use mprotect to commit.

Homey

Yes. The C library / kernel effectively implicitly does this anyway when you allocate a chunk of memory of any size normally. Pages are reserved but not actually mapped until you touch them. Mapping /dev/zero (with MAP_PRIVATE) is just a special kind of malloc. – Inattention

@MarkR: That's true for linux, for certain settings of the vm overcommit sysctl. The mmap of /dev/zero approach ought to work on any POSIX OS including linux with vm overcommit disabled. Reading the vm overcommit documentation, the combination of /dev/zero and no write access is what causes the block to not count against the commit limit. – Homey

Answer 2

You can turn this functionality on system-wide by using kernel overcommit. This is usually default setting on many distributions.

Here is the explanation http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting

Enchain

Oh that is cool... Can't turn it on for only a single process though huh? – Eldwon

@dlamotte: It's typically on by default on a desktop system. Real-time systems would frown upon this sort of behavior (and demand-paging in general), but it's beneficial on desktops and I haven't come across one that doesn't already have that turned on. As long as you never write to the memory you allocate (through almost any means) the kernel's only going to reserve it. – Lagerkvist

Answer 3

The Linux equivalent of VirtualAlloc() is mmap(), which provides the same behaviours. However as a commenter points out, reservation of contiguous memory is the behaviour of calls to malloc() as long as the memory is not initialized (such as by calloc(), or user code).

Bighead

You're right, I was surprised to find that it is specified in the docs for malloc that you linked. – Alburnum

By the way, with only malloc I was able to allocate just shy of the 128TB of virtual address space on a linux x64 machine, in 4GB chunks (2^15-3 or so) with about 500MB overhead (too low in fact for there to have been a page table entry for each 4kb page even, but just right for an 8 byte entry for each 2MB page (transparent huge pages support in the kernel?) – Alburnum

Answer 4

"seemingly random unacceptable latency for very large arrays

You could also consider mlock() or mmap() + MAP_LOCKED to mitigate the impact of paging. Many CPUs support huge (aka large) pages, pages larger than 4kb. These larger pages can mitigate the impact of the TLB on streaming reads/writes.

Idio

The problem is still the copy, even without paging it's too slow to copy a 2GB array to make a 4GB one. If you can avoid moving the memory you won't have that latency hit on adding the 2 billionth and 1 element. – Alburnum