How do I know if my server has NUMA?

Question

Hopping from Java Garbage Collection, I came across JVM settings for NUMA. Curiously I wanted to check if my CentOS server has NUMA capabilities or not. Is there a *ix command or utility that could grab this info?

Babi

Answer 1

I'm no expert here, but here's something:

Box 1, no NUMA:

~$ dmesg | grep -i numa
[    0.000000] No NUMA configuration found

Box 2, some NUMA:

~$ dmesg | grep -i numa
[    0.000000] NUMA: Initialized distance table, cnt=8
[    0.000000] NUMA: Node 4 [0,80000000) + [100000000,280000000) -> [0,280000000)

Fricandeau

Mine doesn't even say "No NUMA configuration", matches at all (linux 2.6.18 / centos)... – Discrepancy

dmesg for me also lacks any mention of "NUMA", because it's too early. grep /var/log/dmesg instead, as it is more likely to have the complete log. (And what does "NUMA turned off." mean?) – Hite

For me, dmesg also lacks mention of "NUMA", but I have no access to /var/log/dmesg which need root privilege. I run find /proc|grep -i numa and saw some numa_maps files. I guess this is also a symbol that NUMA is enabled. – Hydrograph

The approach of checking for Numa related messages in dmesg is unreliable. Firstly, since dmesg reads the kernel ring buffer, the message may have gone by the time you grep for it. i.e. False negative. You can't distinguish this from the case where the kernel is either so early, or sufficiently stripped down that Numa messages are completely absent. – Carob

I never said it's reliable. It is just one of the tools. – Fricandeau

Answer 2

I think this previous question is similar: How to confirm NUMA?

In particular, you can review the NUMA man page here: http://man7.org/linux/man-pages/man7/numa.7.html

And from there you'll see:

$ find /proc -name numa_maps
/proc/1/task/1/numa_maps
/proc/1/numa_maps
/proc/2/task/2/numa_maps
/proc/2/numa_maps
/proc/3/task/3/numa_maps
[etc if you have numa]

And you can get more detail like so:

$ grep NUMA=y /boot/config-`uname -r`
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_ACPI_NUMA=y

$ numactl --hardware
available: 2 nodes (0-1)
node 0 size: 18156 MB
node 0 free: 9053 MB
node 1 size: 18180 MB
node 1 free: 6853 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

Carob

Answer 3

You can also get this info from lscpu command:

lscpu | grep -i numa
NUMA node(s):          2
NUMA node0 CPU(s):     0-19,40-59
NUMA node1 CPU(s):     20-39,60-79

Spy

Answer 4

For Redhat 4,5,6 and 7 systems, one can try the following to determine if NUMA configuration is disabled:

numactl --show does not show multiple nodes

# numactl --show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpubind: 0
nodebind: 0
membind: 0

or numactl --hardware does not list multiple nodes

# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
node 0 size: 524163 MB
node 0 free: 505253 MB
node distances:
node   0
  0:  10

Jollity

Also available on a modern Ubuntu distro with numactl package. – Sayette

Answer 5

You can also just query the information from /sys (this is what tools like numactl do underneath). As others pointed out, using dmesg will be unreliable since it usually does not have unlimited buffering.

To find out how many NUMA nodes are currently available, do:

cat /sys/devices/system/node/online
0-3

Georgenegeorges

Hi @Georgenegeorges ! Do you know if there is documentation with more in depth details related to files tree /sys/devices/system/node/. I looked for it in kernel docs but unfortunately there is no documentation related to this area. I would like to know what stats are stored in specific files in that dir. – Sizemore

It's documented under kernel.org/doc/Documentation/ABI/stable/sysfs-devices-node – Georgenegeorges