Using mmap and madvise for huge pages

Question

I want to allocate memory on the hugepages being used by a Linux machine. I see that there are two ways to do this, using mmap and madvise.

That is, using the MAP_HUGETLB flag with the mmap call -

base_ptr_ = mmap(NULL, memory_size_, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);

And the MADV_HUGEPAGE flag with the madvise call -

madvise(base_ptr_, memory_size_, MADV_HUGEPAGE);

Could someone explain the difference between the two?

Region

Answer 1

Both functions perform different operations, which may or may not matter in your context:

• madvise sets a flag for all the memory mappings corresponding to the region passed to it, telling the khugepaged kernel thread that it can consider said mappings for promotion to huge pages. That will only work if transparent hugepage support is enabled (The status of transparent hugepage support is available under /sys/kernel/mm/transparent_hugepage/enabled), which will be the case in most distros, but may be disabled on embedded systems.

• mmap will actually go ahead and reserve the pages from the kernel's internal hugetlbfs mount, whose status can be seen under /sys/kernel/mm/hugepages. The pages in question need to be available by the time mmap is invoked (see HugePages_Free in /proc/meminfo), or mmap will fail.

The two mechanisms have their own doc file in the kernel tree: hugetlbpage.txt and transhuge.txt

Undine

So given there are enough pages available... mmap is the more reliable option? Is that correct? – Region

It depends, if you control everything that's on the machine, and know exactly how your program will behave, then mmap will allow you to specify exactly how to behave. If you're unsure about how the memory will be used (will it fragment? are there other huge page users in the system), then transparent hugepage will give you the best possible outcome eventually. – Undine

also, madvise is only applicable to private anonymous mappings. – Perfectionism

Answer 2

mmap is for Explicit Hugepages while madvise is for Transparent Hugepages. These are two ways of using large pages on linux. Whilst EHP are reserved in virtual memory upfront, THPs are not

Provinciality