Trying to locate a leak! What does anon mean for pmap?

Question

I'm trying to locate where my memory has gone for a java process running in linux. Someone suggested I use pmap -x to see exactly what the memory is doing.

The output is really long but basically a good portion of it is a repeat of this:

00007fbf75f6a000    1016       -       -       - rwx--    [ anon ]
00007fbf76068000      12       -       -       - -----    [ anon ]

What exactly does this mean? Why do I have so many entries of this (4000+)?

Daliladalis

Answer 1

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block).

In my experience, thread stacks also use anon blocks. If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause. Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading.

b089f000    504K rwx--    [ anon ]
b091d000     12K -----    [ anon ]
b0920000    504K rwx--    [ anon ]
b099e000     12K -----    [ anon ]
b09a1000    504K rwx--    [ anon ]
b0a1f000     12K -----    [ anon ]

But that leaves a question: why are you using pmap to diagnose a Java memory issue?

Debug

> why are you using pmap to diagnose a Java memory issue? We have a java process with heap set to max 256MB but it's RSS memory is 8.9GB. What other tool can we use to diagnose this? – Psychasthenia

@Psychasthenia - sure, makes sense for you. But are you the OP? (and if yes, why are you commenting 8 years later?). From the original question, it was unclear whether the OP knew what he/she was looking for. – Debug

And as a suggestion: if you're seeing such a disparity between configured heap size and RSS, look for memory-mapped files or direct buffers. You'll see such behavior, for example, from a Kafka or SOLR server, and it's normal. – Debug

Answer 2

See this part this part of System Performance Tuning for anonymous memory.

Stowell

That tells you how anonymous allocations are managed, not what it is. ALthough the link overall is quite a good one. – Debug

Answer 3

I've seen that pattern before in a thread leak. If you have code that is trying to pool threads, but somehow messes up and leaks a thread, you get a pattern like that in pmap.

I think each bit of memory is the minimum stack size for the thread, certainly it had nothing to do with heap in our case.
We still got OutOfMemoryErrors when we hit OS limits, even tho when we analise the heap it is not overallocated.

When we had a problem like this pmap [pid] | grep -c 12K turned out to be the number of threads in use.

Infinite

Answer 4

Use Eclipse MAT (when you get OutOfMemoryExceptions in the Java Heap not the native heap).

Take