why java process use much more memory than expected

Question

SystemInfo

OS: archlinux

JDK: OpenJDK (IcedTea 2.4.3) (ArchLinux build 7.u45_2.4.3-1-x86_64)

Running App: https://github.com/aemoncannon/ensime

My java options:

-Xms128m -Xmx512M -Xss1M -XX:MaxPermSize=50m 
-XX:MaxDirectMemorySize=10M  
-XX:ReservedCodeCacheSize=64m 
-Dfile.encoding=UTF-8 -XX:NativeMemoryTracking=detai

Linux top command shows, It uses 6.8G * 19.5% = 1.326GB

Natvie memory tracking shows it use 713MB

So. Which one is the actually physic memory usage, and if it use 1.3GB memory, where does the extra 600+MB physic memory go?

Top outputs

top - 09:20:37 up 48 min,  2 users,  load average: 0.31, 0.33, 0.36
Tasks: 161 total,   1 running, 159 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 
KiB Mem:   7091684 total,  4338428 used,  2753256 free,   117044 buffers
KiB Swap:  8388604 total,        0 used,  8388604 free,  1399904 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  
 5666 jilen     20   0 3553516 1.317g  22652 S   0.3 19.5   2:27.72 java 

The native memory tracking outpus(jcmd VM.native_memory detail)

Native Memory Tracking:

Total:  reserved=731125KB,  committed=681117KB

-                 Java Heap (reserved=575488KB, committed=574976KB)
                            (mmap: reserved=575488KB, committed=574976KB)

-                     Class (reserved=6388KB, committed=6388KB)
                            (classes #6443)
                            (malloc=6388KB, #16725)

-                    Thread (reserved=27986KB, committed=27986KB)
                            (thread #27)
                            (stack: reserved=27756KB, committed=27756KB)
                            (malloc=82KB, #114)
                            (arena=148KB, #54)

-                      Code (reserved=70523KB, committed=21059KB)
                            (malloc=3963KB, #6149)
                            (mmap: reserved=66560KB, committed=17096KB)

-                        GC (reserved=26964KB, committed=26964KB)
                            (malloc=5900KB, #158)
                            (mmap: reserved=21064KB, committed=21064KB)

-                  Compiler (reserved=176KB, committed=176KB)
                            (malloc=78KB, #238)
                            (arena=98KB, #2)

-                  Internal (reserved=5463KB, committed=5431KB)
                            (malloc=5431KB, #1492)
                            (mmap: reserved=32KB, committed=0KB)

-                    Symbol (reserved=10668KB, committed=10668KB)
                            (malloc=9765KB, #77949)
                            (arena=903KB, #1)

-           Memory Tracking (reserved=5616KB, committed=5616KB)
                            (malloc=5616KB, #120)

-        Pooled Free Chunks (reserved=1854KB, committed=1854KB)
                            (malloc=1854KB)


Virtual memory map:

[0x00000000dce00000 - 0x0000000100000000] reserved 575488KB for Java Heap
        from [JVM_handle_linux_signal+0x173dbd]
    [0x00000000dce00000 - 0x00000000fff80000] committed 574976KB from [JVM_handle_linux_signal+0x650ab]

[0x00007f53801c7000 - 0x00007f53802c8000] reserved 1028KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f53801c7000 - 0x00007f53802c8000] committed 1028KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f5381daf000 - 0x00007f5381eb0000] reserved 1028KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f5381daf000 - 0x00007f5381eb0000] committed 1028KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f53821f0000 - 0x00007f53824f3000] reserved 3084KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f53821f0000 - 0x00007f53824f3000] committed 3084KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f5382d49000 - 0x00007f5382e4a000] reserved 1028KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f5382d49000 - 0x00007f5382e4a000] committed 1028KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f538302a000 - 0x00007f538342e000] reserved 4112KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f538302a000 - 0x00007f538342e000] committed 4112KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f53b47c7000 - 0x00007f53b4aca000] reserved 3084KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f53b47c7000 - 0x00007f53b4aca000] committed 3084KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f53b4db7000 - 0x00007f53b4eb8000] reserved 1028KB for Thread Stack
        from [JVM_handle_linux_signal+0x127be9]
    [0x00007f53b4db7000 - 0x00007f53b4eb8000] committed 1028KB from [JVM_handle_linux_signal+0x127be9]

[0x00007f53b4eb8000 - 0x00007f53b52bc000] reserved 4112KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f53b4eb8000 - 0x00007f53b52bc000] committed 4112KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f53b5894000 - 0x00007f53b5a96000] reserved 2056KB for Thread Stack
        from [JVM_handle_linux_signal+0x12f2ec]
    [0x00007f53b5894000 - 0x00007f53b5a96000] committed 2056KB from [JVM_handle_linux_signal+0x12f2ec]

[0x00007f53b5a96000 - 0x00007f53b5b97000] reserved 1028KB for Thread Stack
        from [JVM_handle_linux_signal+0x17c90d]
    [0x00007f53b5a96000 - 0x00007f53b5b97000] committed 1028KB from [JVM_handle_linux_signal+0x17c90d]

[0x00007f53b5b97000 - 0x00007f53b5c98000] reserved 1028KB for Thread Stack
        from [JVM_FindSignal+0x118f1b]
    [0x00007f53b5b97000 - 0x00007f53b5c98000] committed 1028KB from [JVM_FindSignal+0x118f1b]

[0x00007f53b6e70000 - 0x00007f53b8000000] reserved 17984KB for GC
        from [JVM_handle_linux_signal+0x172529]
    [0x00007f53b6e70000 - 0x00007f53b8000000] committed 17984KB from [JVM_handle_linux_signal+0x650ab]

[0x00007f53c4000000 - 0x00007f53c8000000] reserved 65536KB for Code
        from [JVM_handle_linux_signal+0x1741a5]
    [0x00007f53c4000000 - 0x00007f53c5070000] committed 16832KB from [JVM_handle_linux_signal+0x1746f4]

[0x00007f53cc0e4000 - 0x00007f53cc1fd000] reserved 1124KB for GC
        from [JVM_handle_linux_signal+0x172529]
    [0x00007f53cc0e4000 - 0x00007f53cc1fd000] committed 1124KB from [JVM_handle_linux_signal+0x650ab]

[0x00007f53cc1fd000 - 0x00007f53cc601000] reserved 4112KB for Thread Stack
        from [AsyncGetCallTrace+0x3949c]
    [0x00007f53cc1fd000 - 0x00007f53cc601000] committed 4112KB from [AsyncGetCallTrace+0x3949c]

[0x00007f53cc601000 - 0x00007f53cc71b000] reserved 1128KB for GC
        from [JVM_handle_linux_signal+0x172529]
    [0x00007f53cc601000 - 0x00007f53cc71b000] committed 1128KB from [dlerror+0x3aad37]

[0x00007f53cc71b000 - 0x00007f53cc81b000] reserved 1024KB for Code
        from [JVM_handle_linux_signal+0x172289]
    [0x00007f53cc71b000 - 0x00007f53cc75d000] committed 264KB from [JVM_handle_linux_signal+0x1746f4]

[0x00007f53cf497000 - 0x00007f53cf4a2000] reserved 44KB for GC
        from [JVM_handle_linux_signal+0x172529]
    [0x00007f53cf497000 - 0x00007f53cf54d000] committed 728KB from [JVM_handle_linux_signal+0x650ab]

[0x00007f53cf4a2000 - 0x00007f53cf54d000] reserved 684KB for GC
        from [JVM_handle_linux_signal+0x172289]

[0x00007f53cf54d000 - 0x00007f53cf64e000] reserved 1028KB for Thread Stack
        from [JVM_handle_linux_signal+0x12c997]
    [0x00007f53cf54d000 - 0x00007f53cf64e000] committed 1028KB from [JVM_handle_linux_signal+0x12c997]

[0x00007f53cf656000 - 0x00007f53cf66f000] reserved 100KB for GC
        from [JVM_handle_linux_signal+0x172289]
    [0x00007f53cf656000 - 0x00007f53cf66f000] committed 100KB from [JVM_handle_linux_signal+0x650ab]

[0x00007f53cf66f000 - 0x00007f53cf677000] reserved 32KB for Internal
        from [JVM_handle_linux_signal+0x3c924]

Details:

[0x00007f53ce330855] JVM_handle_linux_signal+0xee085
                            (malloc=6627KB #77628)

[0x00007f53ce291061] JVM_handle_linux_signal+0x4e891
                            (malloc=5120KB #5)

[0x00007f53ce293021] JVM_handle_linux_signal+0x50851
                            (malloc=5120KB #5)

[0x00007f53ce2a19b5] JVM_handle_linux_signal+0x5f1e5
                            (malloc=5120KB #5)

[0x00007f53ce1c8c64] JVM_FindSignal+0x1188f4
                            (malloc=4040KB #1)

[0x00007f53cde99851] dlerror+0x409851
                            (malloc=3903KB #5678)

[0x00007f53cdd2dcb3] dlerror+0x29dcb3
                            (malloc=2643KB #148)

[0x00007f53cdff5b86] AsyncGetCallTrace+0x6dc46
                            (malloc=2512KB #314)

[0x00007f53ce1c0b17] JVM_FindSignal+0x1107a7
                            (malloc=1274KB #106)

[0x00007f53ce291149] JVM_handle_linux_signal+0x4e979
                            (malloc=640KB #5)

[0x00007f53ce233375] JVM_FindSignal+0x183005
                            (malloc=524KB #419)

[0x00007f53ce38c670] JVM_handle_linux_signal+0x149ea0
                            (malloc=469KB #1)

[0x00007f53cde87c53] dlerror+0x3f7c53
                            (malloc=408KB #16085)

[0x00007f53cdd2d142] dlerror+0x29d142
                            (malloc=360KB #1)

[0x00007f53cdff5186] AsyncGetCallTrace+0x6d246
                            (malloc=310KB #25)

[0x00007f53ce38c581] JVM_handle_linux_signal+0x149db1
                            (malloc=156KB #1)

[0x00007f53cdd38523] dlerror+0x2a8523
                            (malloc=127KB #681)

[0x00007f53ce1c5acb] JVM_FindSignal+0x11575b
                            (malloc=96KB #2)

[0x00007f53ce1c807e] JVM_FindSignal+0x117d0e
                            (malloc=96KB #1)

[0x00007f53ce38e0e4] JVM_handle_linux_signal+0x14b914
                            (malloc=80KB #5)

[0x00007f53ce368abd] JVM_handle_linux_signal+0x1262ed
                            (malloc=71KB #25)

[0x00007f53ce1b4c27] JVM_FindSignal+0x1048b7
                            (malloc=64KB #16)

[0x00007f53ce1b4c97] JVM_FindSignal+0x104927
                            (malloc=64KB #16)

[0x00007f53ce25c867] JVM_handle_linux_signal+0x1a097
                            (malloc=55KB #116)

[0x00007f53ce1c0584] JVM_FindSignal+0x110214
                            (malloc=51KB #1)

[0x00007f53ce1c5a15] JVM_FindSignal+0x1156a5
                            (malloc=48KB #1)

[0x00007f53cdea0fc3] dlerror+0x410fc3
                            (malloc=37KB #464)

[0x00007f53cdf169bd] dlerror+0x4869bd
                            (malloc=36KB #13)

[0x00007f53ce3b2aa7] JVM_handle_linux_signal+0x1702d7
                            (malloc=24KB #13)

[0x00007f53cdff6496] AsyncGetCallTrace+0x6e556
                            (malloc=20KB #4)

[0x00007f53cdeb2ff3] dlerror+0x422ff3
                            (malloc=17KB #180)

[0x00007f53ce27ba1b] JVM_handle_linux_signal+0x3924b
                            (malloc=15KB #255)

[0x00007f53ce368b9d] JVM_handle_linux_signal+0x1263cd
                            (malloc=10KB #1)

[0x00007f53ce367fdb] JVM_handle_linux_signal+0x12580b
                            (malloc=8KB #47)

[0x00007f53cdff2df5] AsyncGetCallTrace+0x6aeb5
                            (malloc=8KB #11)

[0x00007f53cdf5d7d4] dlerror+0x4cd7d4
                            (malloc=8KB #1)

[0x00007f53ce289bc4] JVM_handle_linux_signal+0x473f4
                            (malloc=8KB #1)

[0x00007f53ce2059d6] JVM_FindSignal+0x155666
                            (malloc=7KB #62)

[0x00007f53ce27bb04] JVM_handle_linux_signal+0x39334
                            (malloc=7KB #251)

[0x00007f53ce242453] fork1+0x52b3
                            (malloc=6KB #28)

[0x00007f53ce024918] AsyncGetCallTrace+0x9c9d8
                            (malloc=5KB #18)

[0x00007f53cdd2d333] dlerror+0x29d333
                            (malloc=2KB #57)

[0x00007f53ce36a598] JVM_handle_linux_signal+0x127dc8
                            (malloc=2KB #1)

[0x00007f53ce2d6e24] JVM_handle_linux_signal+0x94654
                            (malloc=2KB #1)

[0x00007f53ce290d96] JVM_handle_linux_signal+0x4e5c6
                            (malloc=2KB #5)

[0x00007f53ce2a1536] JVM_handle_linux_signal+0x5ed66
                            (malloc=2KB #5)

[0x00007f53ce25a49b] JVM_handle_linux_signal+0x17ccb
                            (malloc=2KB #7)

[0x00007f53cde881eb] dlerror+0x3f81eb
                            (malloc=1KB #15)

[0x00007f53ce171720] JVM_FindSignal+0xc13b0
                            (malloc=1KB #74)

[0x00007f53cdf5ef54] dlerror+0x4cef54
                            (malloc=1KB #1)

[0x00007f53cde8a8aa] dlerror+0x3fa8aa
                            (malloc=1KB #59)

[0x00007f53ce3b658d] JVM_handle_linux_signal+0x173dbd
                            (mmap: reserved=575488KB, committed=574976KB)

[0x00007f53ce3b6975] JVM_handle_linux_signal+0x1741a5
                            (mmap: reserved=65536KB, committed=16832KB)

[0x00007f53ce3b4cf9] JVM_handle_linux_signal+0x172529
                            (mmap: reserved=20280KB, committed=20280KB)

[0x00007f53ce371abc] JVM_handle_linux_signal+0x12f2ec
                            (mmap: reserved=19532KB, committed=19532KB)

[0x00007f53cdfc13dc] AsyncGetCallTrace+0x3949c
                            (mmap: reserved=4112KB, committed=4112KB)

[0x00007f53ce3b4a59] JVM_handle_linux_signal+0x172289
                            (mmap: reserved=1808KB, committed=1048KB)

[0x00007f53ce1c928b] JVM_FindSignal+0x118f1b
                            (mmap: reserved=1028KB, committed=1028KB)

[0x00007f53ce36a3b9] JVM_handle_linux_signal+0x127be9
                            (mmap: reserved=1028KB, committed=1028KB)

[0x00007f53ce36f167] JVM_handle_linux_signal+0x12c997
                            (mmap: reserved=1028KB, committed=1028KB)

[0x00007f53ce3bf0dd] JVM_handle_linux_signal+0x17c90d
                            (mmap: reserved=1028KB, committed=1028KB)

[0x00007f53ce27f0f4] JVM_handle_linux_signal+0x3c924
                            (mmap: reserved=32KB, committed=0KB)

Feria

Native memory tracking tracks use of virtual memory. RES and %MEM track use of physical memory. The two have almost nothing to do with each other. The answer to which one is actually memory usage depends on whether by "memory usage" you mean physical memory usage or virtual memory usage. – Orgell

@DavidSchwartz I mean physic memory usage, sorry! – Feria

Answer 1

The RES or %MEM value is the actual usage of physical memory.

Because your system is under no memory pressure at all, no attempt is made to recover physical memory. This is because such an effort is a losing proposition. If the memory is not soon used for some other purpose, the effort to make it free is wasted. If the memory is used soon for some other purpose, the effort to make it free is worse than wasted, it has to be undone to use the memory again. Modern operating systems directly transition memory from one use to another when memory is needed without having to make that memory free. Memory is only made free when there is no other choice.

So, basically, the physical memory is temporarily being wasted until the system can find some better use for it. If you run some other tasks on the machine that consume physical memory, you will see the RES and %MEM values drop as the physical memory is transitioned to another use.

Orgell

But I have set the Xmx and DirectMemorySize, Why does the process use twice as much more as I set – Feria

Those control usage of virtual memory, not physical memory. They have an indirect affect on physical memory usage, but they don't control it directly. Your OS is just being smart and not engaging in pointless effort. Put it under some memory pressure and watch the resident set shrink. – Orgell

Do you mean some of those memory are actually free, but not revoked by system? – Feria

@Feria They're not free, but they could be freed and the process would run just the same. It makes no difference whether they are free or not, except that right now they are not free, and so it would take effort to make them free. That effort would have no payoff, since the system is under no memory pressure. So the OS is smart and just leaves it. – Orgell

Confirmed I use chrome to open about 100 tabs , then memory usage goes down to 12%. Thanks again!! – Feria