Is there a size limit for kernel module in linux?

Question

I have a problem loading a kernel module, there is a large data structure, around the size of 2Gb of memory - whether I preallocate the table (so that it shows in .bss when I do size -A module.ko or try to vmalloc() it at load time, the module loading fails with insmod: error inserting 'module.ko': -1 Cannot allocate memory.

I tried debugging the problem on usermode linux, but I get a bunch of segfaults (that can be continued in gdb, but end up with a console message overflow in relocation type 10 val <value in the ball park of 6G> and 'module' likely not compiled with -mcmodel=kernel. I assume that with Kbuild the -mcmodel should be right, right?

So the questions are:

1. Is there a generic 2G limit for linux kernel module size?
2. Is there a specific 2G limit for kernel modules in usernode linux (I think that in past I've noticed that a large kernel module needs a clean, continuous block of memory...)
3. Can I specify -mcmodel=large for a kernel module and expect it to work?

I've tried this on debian squeeze, 64-bit, 2.6.32-5-amd64 (host) with 8Gb of memory and 2.6.32 in uml with 4G memory, so this should not be an ordinary out of memory problem.

Extra credit for working around the limit, if such limit exists :)

Pubilis

2GB code + data segments? You gotta be kidding! Can't you just mmap your data and not compile it in? – Equisetum

Unfortunately, I kid you not. – Pubilis

Answer 1

If I define the table as static - the module loading will indeed fail - this is probably because of the 1.5G limit mentioned in the answer by Andrew Aylett

However, if I do dynamic vmalloc() calls, I was able to get up to 7680Mb on a host with 8Gb of memory (until the kernel killed some crucial process and my X hanged).

So to answer my questions:

1. Yes, but only for data that is compiled in as static
2. Doesn't look like it.
3. There is no need to do that.

Extra credit: just do vmalloc()

This only works in linux kernels newer than 2.6.10 - before that, the vmalloc() limit was 64 Mb.

Pubilis

Answer 2

As for your first question - the limit on the module itself is 64 megabytes. The module loader will reject to load a module that exceeds this size. From kernel/module.c:

if (len > 64 * 1024 * 1024 || (hdr = vmalloc(len)) == NULL)
        return ERR_PTR(-ENOMEM);

This is true both for 2.6.32 and for the newer kernels too, up to 3.3.

EDIT: In kernel version 3.4, the 64 Mb limit was removed. Now the actual limit depends only on how much memory vmalloc() can allocate.

Jannjanna

Answer 3

Remember that kernel-space memory is different from user-space memory -- on 32-bit Linux, the kernel has only 1Gb address space. There's a log more space address space for the kernel on 64-bit Linux, but kernel documentation suggests that only 1536MB is available for modules.

Glucinum

On 64-bit, there is essentially no address space limit (well, the limit is many TB). – Rossman

As for x86-64, probably this brief description of memory areas in the kernel space may help: Documentation/x86/x86_64/mm.txt. The upper limits can be deduced from there, although the actual limits can be lower of course. – Jannjanna

If I do a dynamic vmalloc(), it seems I can get as much memory as I want (and is available) - see my test code at paste.pocoo.org/show/406020 – Pubilis