redis bgsave failed because fork Cannot allocate memory

Question

all: here is my server memory info with 'free -m'

              total       used       free     shared    buffers     cached
 Mem:         64433       49259      15174          0          3         31
 -/+ buffers/cache:      49224      15209
 Swap:         8197        184       8012

my redis-server has used 46G memory, there is almost 15G memory left free

As my knowledge,fork is copy on write, it should not failed when there has 15G free memory,which is enough to malloc necessary kernel structures .

besides, when redis-server used 42G memory, bgsave is ok and fork is ok too.

Is there any vm parameter I can tune to make fork return success ?

Kaiserism

Get double more memory – Sheeree

Answer 1

From proc(5) man pages:

/proc/sys/vm/overcommit_memory

This file contains the kernel virtual memory accounting mode. Values are:

0: heuristic overcommit (this is the default)

1: always overcommit, never check

2: always check, never overcommit

In mode 0, calls of mmap(2) with MAP_NORESERVE set are not checked, and the default check is very weak, leading to the risk of getting a process "OOM-killed". Under Linux 2.4 any non-zero value implies mode 1. In mode 2 (available since Linux 2.6), the total virtual address space on the system is limited to (SS + RAM*(r/100)), where SS is the size of the swap space, and RAM is the size of the physical memory, and r is the contents of the file /proc/sys/vm/overcommit_ratio.

Emanative

Answer 2

More specifically, from the Redis FAQ

Redis background saving schema relies on the copy-on-write semantic of fork in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages, with the result that if you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.

Setting overcommit_memory to 1 says Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.

Redis doesn't need as much memory as the OS thinks it does to write to disk, so may pre-emptively fail the fork.

Kashakashden

Answer 3

Modify /etc/sysctl.conf and add:

vm.overcommit_memory=1

Then restart sysctl with:

On FreeBSD:

sudo /etc/rc.d/sysctl reload

On Linux:

sudo sysctl -p /etc/sysctl.conf

Lowestoft

Instead of reloading, you can also do (at least in debian): sysctl -w vm.overcommit_memory=1 and then back with sysctl -w vm.overcommit_memory=0 after the backup is done – Aenneea

Answer 4

Redis's fork-based snapshotting method can effectively double physical memory usage and easily OOM in cases like yours. Reliance on linux virtual memory for doing snapshotting is problematic, because Linux has no visibility into Redis data structures.

Recently a new redis-compatible project Dragonfly has been released. Among other things, it solves the OOM problem entirely. (disclosure - I am the author of this project).

Rriocard