Prometheus reports a different value for node_memory_Active_bytes and free -b
Asked Answered
P

2

9

I'm quite new to Prometheus, so this is possibly a silly question - but here goes.

When I request node_memory_Active_bytes and node_memory_MemTotal_bytes I get:

node_memory_Active_bytes{instance="10.1.4.9:9100",job="node-exporter"}  3257815040

(..and..)

node_memory_MemTotal_bytes{instance="10.1.4.9:9100",job="node-exporter"}    16509550592

However, on the box, if I run free -b, I get the following:

$ free -b
              total        used        free      shared  buff/cache   available
Mem:    16509550592  2264915968  6787731456    59121664  7456903168 14140530688
Swap:             0           0           0

I'm struggling to maps prometheus's view of memory with the actual machine's view. I'm sure I must be doing something wrong, but I have no idea what. The total's match, but the free

Picker answered 14/1, 2020 at 17:26 Comment(0)
A
33

Equivalent queries for free -b command:

  • total: node_memory_MemTotal_bytes
  • used: node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes - node_memory_SReclaimable_bytes
  • shared: node_memory_Shmem_bytes
  • free: node_memory_MemFree_bytes
  • buff/cache: node_memory_Buffers_bytes + node_memory_Cached_bytes + node_memory_SReclaimable_bytes
  • available: node_memory_MemAvailable_bytes
Aves answered 15/3, 2021 at 20:6 Comment(3)
You save my day! This should be the right answer.Jorry
The used value from free correlates with: node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes Not as stated by this answer: node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes - node_memory_SReclaimable_bytesLoner
The man page even states Used or unavailable memory (calculated as total - available) man7.org/linux/man-pages/man1/free.1.htmlLoner
C
9

I think you're confusing "active" memory with "used" memory, aren't you? They're different things.

See more info about this here.

To get "used" memory with Prometheus Node Exporter calculate:

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
Carrara answered 14/1, 2020 at 18:4 Comment(3)
Ahh! Makes sense, it's still a bit out though. node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes = 2394025984 used = 2282270720 Any reason for that? This happens consistently over a fair sample of time.Picker
Uhm... I don't know :-(Kevinkevina
The discrepancy is possibly due to the estimated nature of available. From the man page: Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use (MemAvailable in /proc/meminfo, available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free)Loner

© 2022 - 2024 — McMap. All rights reserved.