Increase maximum virtual memory size above 256gb

Question

I'm running a program which allocates 8mb stacks using mmap. While testing to see how many stacks I could allocate (aiming for 100,000), I see virtual memory size rise quickly as expected, and reserved size stay small (less than 1gb). The program then segfaults with Cannot allocate new fiber stack: Cannot allocate memory (Errno). Using gdb to rescue the segfault and then looking at htop, I have discovered this happens at around 256GB of virtual memory.

I've tried using prlimit --as=unlimited --rss=unlimited --memlock=unlimited --data=unlimited when running the program, but it doesn't seem to make a difference.

Is there a way to increase this limit? Is it advisable to increase this limit? Is there a better way for crystal to allocate stacks?

Intercontinental

Answer 1

Maybe you're hitting the maximum of /proc/sys/vm/max_map_count. This setting sets a maximum on the number of mmaps your process can have. The default value is 65536. So it's likely not the size of memory you want to malloc, but the number of malloc calls that causes the error Cannot allocate memory.

You can try to increase the maximum with:

sysctl -w vm.max_map_count=131070

See also NPTL caps maximum threads at 65528?

Lonna

This is indeed the limit, I increased it to 10 million and got over 5 million fibers going in Crystal. Thanks! – Intercontinental

Solved my is in Golang fatal error: out of memory allocating heap arena metadata, where map hashGrow makeBucketArray failed at 256GB on a 1TB RAM EC2 Instance – Numismatics

Answer 2

I'd check your swap file size. if you are running out of swap then all those parameter changes wont help you until you fix that.

I'd recreate the failure and run free -h to see if there is any unused swap. If its all gone you will need to increase your swap size.

Adama

I have 16GB of ram and no swap space, yet I can still allocate 256GB of virtual memory. I don't think swap is the issue here. – Intercontinental