What is the difference between vmalloc and kmalloc?

Question

I've googled around and found most people advocating the use of kmalloc, as you're guaranteed to get contiguous physical blocks of memory. However, it also seems as though kmalloc can fail if a contiguous physical block that you want can't be found.
What are the advantages of having a contiguous block of memory? Specifically, why would I need to have a contiguous physical block of memory in a system call? Is there any reason I couldn't just use vmalloc?
Finally, if I were to allocate memory during the handling of a system call, should I specify GFP_ATOMIC? Is a system call executed in an atomic context?

GFP_ATOMIC
The allocation is high-priority and does not sleep. This is the flag to use in interrupt handlers, bottom halves and other situations where you cannot sleep.

GFP_KERNEL This is a normal allocation and might block. This is the flag to use in process context code when it is safe to sleep.

Tenderize

A good article on vmalloc and kmalloc http://learnlinuxconcepts.blogspot.in/2014/02/linux-memory-management.html – Burmese

That article claims nonsense like: "Generally a 32 bit architecture has 4KB page size and a 64 bit architecture has 8KB page size". I haven't fully read it, but I wouldn't call it "good", or even trust a word from it. – Gratulation

Note (semi-related): vmalloc is faster with Kernel 5.2 (Q2 2019) – Achromic

Can we use GFP_NOWAIT instead of GFP_ATOMIC in interrupt context? What's the actual difference? – Contexture

Answer 1

You only need to worry about using physically contiguous memory if the buffer will be accessed by a DMA device on a physically addressed bus (like PCI). The trouble is that many system calls have no way to know whether their buffer will eventually be passed to a DMA device: once you pass the buffer to another kernel subsystem, you really cannot know where it is going to go. Even if the kernel does not use the buffer for DMA today, a future development might do so.

vmalloc is often slower than kmalloc, because it may have to remap the buffer space into a virtually contiguous range. kmalloc never remaps, though if not called with GFP_ATOMIC kmalloc can block.

kmalloc is limited in the size of buffer it can provide: 128 KBytes^*). If you need a really big buffer, you have to use vmalloc or some other mechanism like reserving high memory at boot.

^*) This was true of earlier kernels. On recent kernels (I tested this on 2.6.33.2), max size of a single kmalloc is up to 4 MB! (I wrote a fairly detailed post on this.) — kaiwan

For a system call you don't need to pass GFP_ATOMIC to kmalloc(), you can use GFP_KERNEL. You're not an interrupt handler: the application code enters the kernel context by means of a trap, it is not an interrupt.

Roryros

I thought system calls were entered by triggering int $0x80? (i.e. an interrupt)? – Tenderize

int $0x80 is a software interrupt, also called a trap. What is meant by interrupt handlers is a hardware interrupt, such as when the user presses a key or moves the moves. – Hamo

System calls are for userspace to kernel space transitions...kmalloc is used in the kernel context only?? – Chetchetah

@FreeMemory: int $0x80 is x86-specific, and then, it's also an old method superseded by sysenter/syscall (on x86). – Tectonic

Thanks for an answer giving insights. So vmalloc allocates page by page the virtual to physical pages. When you said kmalloc never remaps, what do you mean by that? Does the kernel have pre-allocated virtual-physical region for kmalloc(congiuously allocated)? – Tactical

Answer 2

Short answer: download Linux Device Drivers and read the chapter on memory management.

Seriously, there are a lot of subtle issues related to kernel memory management that you need to understand - I spend a lot of my time debugging problems with it.

vmalloc() is very rarely used, because the kernel rarely uses virtual memory. kmalloc() is what is typically used, but you have to know what the consequences of the different flags are and you need a strategy for dealing with what happens when it fails - particularly if you're in an interrupt handler, like you suggested.

Claycomb

"because the kernel rarely uses virtual memory", why is that though? – Help

Because you generally don't want the kernel block while it waits for the kernel to swap memory in or out of disk storage... – Claycomb

No, kernel memory allocated with vmalloc is never swapped. Only userspace memory can be swapped. The kernel address space is unswappable, and vmalloc allocates in the kernel's address space. – Hydromel

Answer 3

The kmalloc() & vmalloc() functions are a simple interface for obtaining kernel memory in byte-sized chunks.

1. The kmalloc() function guarantees that the pages are physically contiguous (and virtually contiguous).

2. The vmalloc() function works in a similar fashion to kmalloc(), except it allocates memory that is only virtually contiguous and not necessarily physically contiguous.

Feeding

Answer 4

Linux Kernel Development by Robert Love (Chapter 12, page 244 in 3rd edition) answers this very clearly.

Yes, physically contiguous memory is not required in many of the cases. Main reason for kmalloc being used more than vmalloc in kernel is performance. The book explains, when big memory chunks are allocated using vmalloc, kernel has to map the physically non-contiguous chunks (pages) into a single contiguous virtual memory region. Since the memory is virtually contiguous and physically non-contiguous, several virtual-to-physical address mappings will have to be added to the page table. And in the worst case, there will be (size of buffer/page size) number of mappings added to the page table.

This also adds pressure on TLB (the cache entries storing recent virtual to physical address mappings) when accessing this buffer. This can lead to thrashing.

Overtrick

Answer 5

What are the advantages of having a contiguous block of memory? Specifically, why would I need to have a contiguous physical block of memory in a system call? Is there any reason I couldn't just use vmalloc?

From Google's "I'm Feeling Lucky" on vmalloc:

kmalloc is the preferred way, as long as you don't need very big areas. The trouble is, if you want to do DMA from/to some hardware device, you'll need to use kmalloc, and you'll probably need bigger chunk. The solution is to allocate memory as soon as possible, before memory gets fragmented.

Cryptology

See, I read that, and it doesn't make sense to me. I understand using kmalloc for big areas; but for small allocations, why not use vmalloc to avoid fragmenting the physical memory? – Tenderize

Because you should trust the kernel to do what's best; if it thinks allocating a single chunk is better, it'll do so. vmalloc is only for when you absolutely must have a contiguous chunk. – Cryptology

I guess that makes sense, but it seems counterintuitive. kmalloc sounds like it ought to be used when performance is of utmost concern (i.e. I can not be plagued by disk IO). Also, what about GFP_ATOMIC? – Tenderize

Answer 6

On a 32-bit system, kmalloc() returns the kernel logical address (its a virtual address though) which has the direct mapping (actually with constant offset) to physical address. This direct mapping ensures that we get a contiguous physical chunk of RAM. Suited for DMA where we give only the initial pointer and expect a contiguous physical mapping thereafter for our operation.

vmalloc() returns the kernel virtual address which in turn might not be having a contiguous mapping on physical RAM. Useful for large memory allocation and in cases where we don't care about that the memory allocated to our process is continuous also in Physical RAM.

Emmalynne

Answer 7

One of other differences is kmalloc will return logical address (else you specify GPF_HIGHMEM). Logical addresses are placed in "low memory" (in the first gigabyte of physical memory) and are mapped directly to physical addresses (use __pa macro to convert it). This property implies kmalloced memory is continuous memory.

In other hand, Vmalloc is able to return virtual addresses from "high memory". These addresses cannot be converted in physical addresses in a direct fashion (you have to use virt_to_page function).

Freeman

Answer 8

In short, vmalloc and kmalloc both could fix fragmentation. vmalloc use memory mappings to fix external fragmentation; kmalloc use slab to fix internal frgamentation. Fot what it's worth, kmalloc also has many other advantages.

Dehlia