How to limit CPU and RAM resources for mongodump?

Question

I have a mongod server running. Each day, I am executing the mongodump in order to have a backup. The problem is that the mongodump will take a lot of resources and it will slow down the server (that by the way already runs some other heavy tasks).

My goal is to somehow limit the mongodump which is called in a shell script.

Thanks.

Stuffing

Keyword you need to google is cgroup. – Ecbolic

@SergeyKlyaus, thanks a lot. Could you please provide an example here as an answer and I will accept it. – Stuffing

Answer 1

You should use cgroups. Mount points and details are different on distros and a kernels. I.e. Debian 7.0 with stock kernel doesn't mount cgroupfs by default and have memory subsystem disabled (folks advise to reboot with cgroup_enabled=memory) while openSUSE 13.1 shipped with all that out of box (due to systemd mostly).

So first of all, create mount points and mount cgroupfs if not yet done by your distro:

mkdir /sys/fs/cgroup/cpu
mount -t cgroup -o cpuacct,cpu cgroup /sys/fs/cgroup/cpu

mkdir /sys/fs/cgroup/memory
mount -t cgroup -o memory cgroup /sys/fs/cgroup/memory

Create a cgroup:

mkdir /sys/fs/cgroup/cpu/shell
mkdir /sys/fs/cgroup/memory/shell

Set up a cgroup. I decided to alter cpu shares. Default value for it is 1024, so setting it to 128 will limit cgroup to 11% of all CPU resources, if there are competitors. If there are still free cpu resources they would be given to mongodump. You may also use cpuset to limit numver of cores available to it.

echo 128 > /sys/fs/cgroup/cpu/shell/cpu.shares
echo 50331648 > /sys/fs/cgroup/memory/shell/memory.limit_in_bytes

Now add PIDs to the cgroup it will also affect all their children.

echo 13065 >  /sys/fs/cgroup/cpu/shell/tasks
echo 13065 >  /sys/fs/cgroup/memory/shell/tasks

I run couple of tests. Python that tries to allocate bunch of mem was Killed by OOM:

myaut@zenbook:~$ python -c 'l = range(3000000)'
Killed

I've also run four infinite loops and fifth in cgroup. As expected, loop that was run in cgroup got only about 45% of CPU time, while the rest of them got 355% (I have 4 cores).

All that changes do not survive reboot!

You may add this code to a script that runs mongodump, or use some permanent solution.

Ecbolic

Do you add the PID of the shell and then the python process that spawns off it inherits the cgroup? – Cronin