Why is LuaJIT's memory limited to 1-2 GB on 64-bit platforms?

Question

On 64-bit platforms, LuaJIT allows only up to 1-2GB of data (not counting objects allocated with malloc). Where does this limitation come from, and why is this even less than on 32-bit platforms?

Kincaid

Can you cite a reference for the 1GB limit? From everything I've seen, the limit is 2GB and the times when others have said it's 1GB for 64-bit they were either dismissed or the threads stopped there – Lecturer

@Lecturer It's 1GB on Linux. I'll clarify – Kincaid

Answer 1

LuaJIT is designed to use 32-bit pointers. On x64 platforms the limit comes from the use of mmap and the MAP_32BIT flag.

MAP_32BIT (since Linux 2.4.20, 2.6):

Put the mapping into the first 2 Gigabytes of the process address space. This flag is supported only on x86-64, for 64-bit programs. It was added to allow thread stacks to be allocated somewhere in the first 2GB of memory, so as to improve context-switch performance on some early 64-bit processors.

Essentially using this flag limits to the first 31-bits, not the first 32-bits as the name suggests. Have a look here for a nice overview of the 1GB limit using MAP_32BIT in the Linux kernel.

Even if you could have more than 1GB, the LuaJIT author explains why this would be bad for performance:

• A full GC takes 50% more time than the allocations themselves.
• If the GC is enabled, it doubles the allocation time.
• To simulate a real application, the links between objects are randomized in the third run. This doubles the GC time!

And that was just for 1GB! Now imagine using 8GB -- a full GC cycle would keep the CPU busy for a whopping 24 seconds! Ok, so the normal mode is to use the incremental GC. But this just means the overhead is ~30% higher, it's mixed in between the allocations and it will evict the CPU cache every time. Basically your application will be dominated by the GC overhead and you'll begin to wonder why it's slow ....

tl;dr version: Don't try this at home. And the GC needs a rewrite (postponed to LuaJIT 2.1).

To summarize, the 1GB limit is a limitation of the Linux kernel and the LuaJIT garbage collector. This only applies to objects within the LuaJIT state and can be overcome by using malloc, which will allocate outside the lower 32-bit address space. Also, it's possible to use the x86 build on x64 in 32-bit mode and have access the full 4GB.

Check out these links for more information:

• How to get past 1gb memory limit of 64 bit LuaJIT on Linux?
• LuaJIT x64 limited to 31 bit address space, even without MAP_32BIT restrictions?
• LuaJIT strange memory limit
• Digging out the craziest bug you never heard about from 2008: a linux threading regression

Amah

Answer 2

Due to recent patch luajit 2GB memory limit can be solved.

To test, clone this repo and build with LUAJIT_ENABLE_GC64 symbol defined:

msvcbuild.bat gc64

or XCFLAGS+= -DLUAJIT_ENABLE_GC64 in Makefile

I used this code to test memory allocation:

local ffi = require("ffi")

local CHUNK_SIZE     = 1 * 1024 * 1024 * 1024
local fraction_of_gb = CHUNK_SIZE / (1024*1024*1024)
local allocations    = {}

for index=1, 64 do
    local huge_memory_chunk = ffi.new("char[?]", CHUNK_SIZE)
    table.insert(allocations, huge_memory_chunk)
    print( string.format("allocated %q GB", index*fraction_of_gb) )
    local pause = io.read(1)
end

print("Test complete")
local pause = io.read(1)

Allocated 48GB before not enough memory error on my machine.

Ventail