ZGC max heap size exceed physical memory

Question

1. JVM options is

-server -Xmx100g -Xms100g -XX:MaxMetaspaceSize=1G -Xss512k 
-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -XX:MaxGCPauseMillis=30 
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=12 
-XX:+DisableExplicitGC -XX:LargePageSizeInBytes=128m

2. RAM is 256G

             total       used       free     shared    buffers     cached
Mem:           251        250          1        100          0        138
-/+ buffers/cache:        112        139
Swap:            7          0          7

3. the top command show the process's RES is 303G

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                           
240731 xxx       20   0 17.0t 302g 297g S  6.6 119.9 256:35.43 java

4. jvm profile show like follow

./jhsdb jmap --heap --pid 240731
Attaching to process ID 240731, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 11.0.2+9

using thread-local object allocation.
ZGC with 12 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 107374182400 (102400.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 17592186044415 MB
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 1073741824 (1024.0MB)
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
 ZHeap          used 82988M, capacity 1024M, max capacity 27112449862M

Any other VM options for the max capatity ? How the ZGC heap works ?

Vizzone

Not enough space available on the backing filesystem to hold the current max Java heap ， why use backing filesystem to hold the current max Java heap ？ How to change options for this – Vizzone

This is a known "problem" with ZGC, that is virtual memory actually. – Micahmicawber

Answer 1

Since this question was asked from time to time, and I am tired of explaining it already, let me try to put it here so people can simply search, get their answer, and be happy again (even for a very short period of time, it's still worth it!).

RSS over-reporting by ZGC is due to the technique that ZGC uses to manipulate the memory pages, namely, multi-mapping. and since ZGC is essentially another implementation of Zing C4 collector (by Azul Systems), Zing also shares the same "over-reporting" RSS issue.

Look into this code:

http://hg.openjdk.java.net/zgc/zgc/file/59c07aef65ac/src/hotspot/os_cpu/linux_x86/zPhysicalMemoryBacking_linux_x86.cpp#l160

void ZPhysicalMemoryBacking::map(ZPhysicalMemory pmem, uintptr_t offset) const {
  if (ZUnmapBadViews) {
    // Only map the good view, for debugging only
    map_view(pmem, ZAddress::good(offset), AlwaysPreTouch);
  } else {
    // Map all views
    map_view(pmem, ZAddress::marked0(offset), AlwaysPreTouch);
    map_view(pmem, ZAddress::marked1(offset), AlwaysPreTouch);
    map_view(pmem, ZAddress::remapped(offset), AlwaysPreTouch);
  }
}

And this:

void ZPhysicalMemoryBacking::map_view(ZPhysicalMemory pmem, uintptr_t addr, bool pretouch) const {
  const size_t nsegments = pmem.nsegments();
  // Map segments
  for (size_t i = 0; i < nsegments; i++) {
    const ZPhysicalMemorySegment segment = pmem.segment(i);
    const size_t size = segment.size();
    const void* const res = mmap((void*)addr, size, PROT_READ|PROT_WRITE, MAP_FIXED|MAP_SHARED, _file.fd(), segment.start());
    if (res == MAP_FAILED) {
      ZErrno err;
      map_failed(err);
    }
    // Advise on use of transparent huge pages before touching it
    if (ZLargePages::is_transparent()) {
      advise_view(addr, size);
    }
    // NUMA interleave memory before touching it
    ZNUMA::memory_interleave(addr, size);
    if (pretouch) {
      pretouch_view(addr, size);
    }
    addr += size;
  }
}

Map all views:

• map_view(pmem, ZAddress::marked0(offset), AlwaysPreTouch);
• map_view(pmem, ZAddress::marked1(offset), AlwaysPreTouch);
• map_view(pmem, ZAddress::remapped(offset), AlwaysPreTouch);

This means that for the same address, ZGC will map it to 3 different views: marked0, marked1 and remapped. These 3 views are reflected int the virtual memory addresses. And this means, 3 different virtual memory addresses will be mapped to the same underlying physical memory, and thus for each physical memory page, there are 3 virtual pages mapped to it.

And if ZGC grows up to be a generational GC (where you have young generation and old generation, instead of one single generation as ZGC is right now), we can expect this number to be increased to 6x of xmx heap size too.

And this is why the multi-mapping used by both Azul Zing and Oracle ZGC put people to panic mode at "top" command. But please note that it is only the virtual memory space that gets reported, so unless your system tool runs into this confusing part, there is no reason for you to call 911.

Kutch

It's not only humans panicking, it's also the Linux OOM killer. How do you suggest we deal with that? – Loreleilorelie

For monitoring: In the newer kernels, you can directly check the /proc/PID/smaps_rollup file: Rss: 640096 kB Pss: 463041 kB ... the second line should be a good estimate of the real memory consumption for any process. To prevent OOM Killer from killing such processes: unix.stackexchange.com/questions/153585/… It has been 3 years after this question, and no more new queries on OOM Killer, so I would assume that OOM Killer is actually smart enough not to kill such processes. – Kutch