Hugepages exist, but are not free nor reserved. Or, how do I free hugepages?

Question

I'm running an SPDK experiment (which uses DPDK, which in turn uses hugepages) and it was working yesterday. I'm running them in a shared enviroment (I think one or two more people use this machine for other stuff). Now, whenever I try to run it, I get a no free hugepages error.

Output of /proc/meminfo is:

HugePages_Total:    1024
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Output of mount:

cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb)

Something that worked on my last environment doesn't work anymore:

umount -a -t hugetlbfs
mount -t hugetlbfs nodev /mnt/huge

Then the output of /proc/meminfo is

HugePages_Total:    1024
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:     1024

But if I try running it:

EAL: No free hugepages reported in hugepages-1048576kB
EAL: No free hugepages reported in hugepages-2048kB
PANIC in rte_eal_init():
Cannot get hugepage information

Why are these pages surplus and not free? Is there any way I can free them? I want to restart the system since there might be other jobs running or people using it.

edit: Restarted the machine, allocated more hugepages and they were free. Executed the test, it crashed and now the hugepages are lost again.

Relevant questions with no working answer (at least for me):

How to release hugepages from the crashed application

How to really free hugepages in Linux for use by a new process?

Tifanie

Answer 1

If you follow the instruction below, you can get rid of the allocated hugepages:

1) Let's check the hugepages which were free at restart

dpdk@dpdkvm:~$ ls /mnt/huge/
empty

dpdk@dpdkvm:~/dpdk-1.8.0/examples/kni$ cat /proc/meminfo
...
HugePages_Total:     256
HugePages_Free:      256
...

2) Starting a dpdk application with wrong parameters, producing an error

dpdk@dpdkvm:~/dpdk-1.8.0/examples/kni$ sudo ./build/kni -c 0x03 -n 2 -- -P -p 0x03 --config="(0,0,1),(1,0,1)"
...
EAL: Error - exiting with code: 1
  Cause: No supported Ethernet device found

3) When I check hugepages, there is not any free

dpdk@dpdkvm:~/dpdk-1.8.0/examples/kni$ cat /proc/meminfo
...
HugePages_Total:     256
HugePages_Free:        0
...

4) Now, when I check the mounted hugepage directory, I can see the files which are not given back to OS by dpdk application.

dpdk@dpdkvm:~/dpdk-1.8.0/examples/kni$ ls /mnt/huge/
...
rtemap_0    rtemap_137  rtemap_176  rtemap_214  rtemap_253  rtemap_62
...

5) Finally, if you remove the files starting with rtemap, you can give the hugepages back

dpdk@dpdkvm:~/dpdk-1.8.0/examples/kni$ sudo rm /mnt/huge/*
[sudo] password for dpdk:
dpdk@dpdkvm:~/dpdk-1.8.0/examples/kni$ cat /proc/meminfo
...
HugePages_Total:     256
HugePages_Free:      256
...

Dayfly

I don't know why I'm not getting this issue anymore, but I remember that the mount point for the hugepages was empty, no files there. If it occurs again I'll try this, thank you! – Tifanie

Actually, there are cases where the mounted directory is empty and yet there are no free hugepages. Umounting and mounting it again worked a few times, now it doesn't anymore and I have no idea why. – Tifanie

also getting this issue with the empty mounts. I think it has something to do with the non-mmap hugepage allocation – Kattiekatuscha