Java G1 garbage collection in production

Question

Since Java 7 is going to use the new G1 garbage collection by default is Java going to be able to handle an order of magnitude larger heap without supposed "devastating" GC pause times? Has anybody actually implemented G1 in production, what were your experiences?

To be fair the only time I have seen really long GC pauses is on very large heaps, much more than a workstation would have. To clarify my question; will G1 open the gateway to heaps in the hundreds of GB? TB?

Nieberg

Although it could be rephrased more specifically, this isn't a horrible question. I really wish people had to explain themselves better than "Not a question" when voting to close. – Donets

I didn't vote to close, but I wished the OP had done a more objective job of detailing his gripes with the current GC. Also, "Java" is a language whereas he is speaking of an implementation, and I don't know what "implementing G1 in production" means, especially with the future tense of the rest of the question. If it is going to be in Java 7, surely no-one has used it in production? – Priapism

@Pascal G1 has been an experimental feature available in the JDK since JDK 6 update 14. By "implementing G1 in production" I think he meant actually using it, is not that hard to figure. And while I agree that G1 is part of JDK 7, not Java, a search for Java 7 on Google returns the JDK 7 homepage as it's first result, and both terms are often used interchangeably. @Benju I wouldn't trust results obtained with G1 on the current JDK as it is experimental, many things could change from now to the official release. – Mutual

It seems JDK 7 including update 1,2 and 3 does not use the G1 gc by default. You can chect it by jinfo -flag UseG1GC pid – Humboldt

Answer 1

It sounds like the point of G1 is to have smaller pause times, even to the point where it has the ability to specify a maximum pause time target.

Garbage collection isn't just a simple "Hey, it's full, let's move everything at once and start over" deal any more--it's fantastically complex, multi-level, background threaded system. It can do much of its maintenance in the background with no pauses at all, and it also uses knowledge of the system's expected patterns at runtime to help--like assuming most objects die right after being created, etc.

I would say GC pause times are going to continue to improve, not worsen, with future releases.

EDIT:

in re-reading it occurred to me that I use Java daily--Eclipse, Azureus, and the apps I develop, and it's been a LONG TIME since I saw a pause. Not a significant pause, but I mean any pause at all.

I've seen pauses when I right-click on windows explorer or (occasionally) when I hook up certain USB hardware, but with Java---none at all.

Is GC still an issue with anyone?

Donets

Agree - the only time I've seen GC pauses is when I've either deliberately or accidentally provoked them with massively parallel garbage-creating code..... – Funky

Yes, GC is still very much an issue when you start dealing with large heaps (>16GB), especially with large tenured generations. – Harpoon

@the-alchemist wow, I've seen your comment in passing a few times and it just struck me that you said 16 GB!! Although I'm absolutely sure you're correct that this can cause huge delays, I want to check that you disabled ALL swapping. On a large memory system, any swapping of java will absolutely kill your system (Because GC is very swap-unfriendly). I'm sure you've already done this, but I just wanted to mention it--because it would make such a huge difference. I've never seen a PC with that much ram--how much do you have? 32g? – Donets

Yes, GCs are problematic for services in that they are what makes it VERY difficult to improve TP99.9 (and higher) limits. Specifically "old generation" GCs can be death traps that all but freeze JVM (and service) for multiple seconds; and for services that typically serve requests in single-digit (or low double-digit) milli seconds this is problematic. For what it's worth this was a practical problem with backend storage used by Amazon's Simple Queue service (can't go into tons of details as it's AWS internal). – Thremmatology

The annoying thing about GC is that Azul invented years ago an ingenious GC algorithm (Azul C4) that can easily cope with hundreds of gigabytes without any noticeable pause times by making very clever use of the processors memory hardware. But nobody knows this and it won't be implemented in the major Java versions soon since it needs some support by the operating system. And the operating system vendors won't do anything until people know about the algorithm and put pressure on the operating system vendors. See azulsystems.com/zing/pgc , managedruntime.org – Laure

More about garbage collection and Azul C4: youtube.com/watch?v=we_enrM7TSY – Libby

@Dr.Hans-PeterStörr I'm assuming you're talking about hardware virtual memory. Actually, OSes already provide interfaces for managing it (mmap and VirtualAlloc), and an application can even use virtualization hardware (ie, nested page tables) to manage it directly. But why JVM devs aren't even thinking of using it is beyond me. Maybe patents? – Bartels

@AleksandrDubinsky Wakend an old thread here. By the way now I'm running on a 64gb machine with a combination of apps that run around 32gb total (and then on top of that Eclipse and a few others). It's still true that the only time I see pauses is when the SYSTEM starts pushing 90% full. – Donets

@AleksandrDubinsky At least in 2010, Azul C4 needed additional kernel functionality. They wanted to contribute that to the Linux kernel, but gave up. Allegedly, the disapproval of Java (and the like) in the Linux kernel community was a factor. At least Gil Tene (Azul CTO) said: "the very idea of enabling something that could make GC better for everyone, on all runtimes, seemed to get people upset and angry. To some people, it seems, GC is only useful for helping lazy people who are too stupid to program without it, and anything that makes it work better should not be encouraged.". – Changeful

@Changeful Is the patch public? (It should be in the mailing list if it was proposed.) I can see the kernel devs hating Java, but I also don't see Azul open sourcing their garbage collector. Patches that only work for some proprietary piece of software are typically maintained by its vendor. A patch like this needs to be made available to and gestate in the community, and generate uptake. I hope the patch is public so I could see what it does. – Bartels

@AleksandrDubinsky Yes, the code that was part of the proposal is public: github.com/GregBowyer/ManagedRuntimeInitiative. C4 is not open source, but at least the rhetoric on Azul's side is that the new kernel functionality can be still useful for other garbage collectors. – Changeful

@Changeful As I suspected, the patch concerns manipulating virtual memory. I wonder why not use virtualization features to do the same thing in userspace. I studied this some more, and I think these page table tricks only optimize compaction (and only of large objects). It doesn't make or break pause-less GC. Pause-less GC revolves around very careful concurrent programming (to allow all GC phases to happen concurrently with program execution). Shenandoah is the OpenJDK attempt at making such a GC. – Bartels

Answer 2

I've been testing it out with a heavy application: 60-70GB allocated to heap, with 20-50GB in use at any time. With these sorts of applications, it's an understatement to say that your mileage may vary. I'm running JDK 1.6_22 on Linux. The minor versions are important-- before about 1.6_20, there were bugs in G1 that caused random NullPointerExceptions.

I've found that it is very good at keeping within the pause target you give it most of the time. The default appears to be a 100ms (0.1 second) pause, and I've been telling it to do half that (-XX:MaxGCPauseMillis=50). However, once it gets really low on memory, it panics and does a full stop-the-world garbage collection. With 65GB, that takes between 30 seconds and 2 minutes. (The number of CPUs probably doesn't make a difference; it's probably limited by the bus speed.)

Compared with CMS (which is not the default server GC, but it should be for web servers and other real-time applications), typical pauses are much more predictable and can be made much shorter. So far I'm having better luck with CMS for the huge pauses, but that may be random; I'm seeing them only a few times every 24 hours. I'm not sure which one will be more appropriate in my production environment at the moment, but probably G1. If Oracle keeps tuning it, I suspect G1 will ultimately be the clear winner.

If you're not having a problem with the existing garbage collectors, there's no reason to consider G1 right now. If you are running a low-latency application, such as a GUI application, G1 is probably the right choice, with MaxGCPauseMillis set really low. If you're running a batch-mode application, G1 doesn't buy you anything.

Pool

Answer 3

Although I have not tested G1 in production, I thought I would comment that GCs are already problematic for cases without "humongous" heaps. Specifically services with just, say, 2 or 4 gigs can be severely impacted by GC. Young generation GCs are usually not problematic as they finish in single-digit milliseconds (or at most double-digit). But old-generation collections are much more problematic as they take multiple seconds with old-gen sizes of 1 gig or above.

Now: in theory CMS can help a lot there, as it can run most of its operation concurrently. However, over time there will be cases where it can not do this and has to fall back to "stop the world" collection. And when that happens (after, say, 1 hour -- not often, but still too often), well, hold on to your f***ing hats. It can take a minute or more. This is especially problematic for services that try to limit maximum latency; instead of it taking, say, 25 milliseconds to serve a request it now takes ten second or more. To add injury to insult clients will then often time out the request and retry, leading to further problems (aka "shit storm").

This is one area where G1 was hoped to help a lot. I worked for a big company that offers cloud services for storage and message dispatching; and we could not use CMS since although much of the time it worked better than parallel varieties, it had these meltdowns. So for about an hour things were nice; and then stuff hit the fan... and because service was based on clusters, when one node got in trouble, others typically followed (since GC-induced timeouts lead to other nodes believe node had crashed, leading to re-routes).

I don't think GC is that much of a problem for apps, and perhaps even non-clustered services are less often affected. But more and more systems are clustered (esp. thanks to NoSQL data stores) and heap sizes are growing. OldGen GCs are super-linearly related to heap size (meaning that doubling heap size more than doubles GC time, assuming size of live data set also doubles).

Thremmatology

Answer 4

We are already using G1GC, from almost two years. Its doing great in our mission critical transaction processing system, and It proved to be a great support w.r.t high throughput, low pauses, concurrency and optimized heavy memory management.

We are using following JVM settings:

-server -Xms512m -Xmx3076m -XX:NewRatio=50 -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:+AggressiveOpts -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=400 -XX:GCPauseIntervalMillis=8000 -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime

Updated

-d64 -server -Xss4m -Xms1024m -Xmx4096m -XX:NewRatio=50 -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:+HeapDumpOnOutOfMemoryError -XX:-DisableExplicitGC -XX:+AggressiveOpts -Xnoclassgc -XX:+UseNUMA -XX:+UseFastAccessorMethods -XX:ReservedCodeCacheSize=48m -XX:+UseStringCache -XX:+UseStringDeduplication -XX:MaxGCPauseMillis=400 -XX:GCPauseIntervalMillis=8000

Mckissick

At Java 8, you don't need to set -XX:+UseCompressedOops or -XX:+DoEscapeAnalysis, booth are on as default. See: docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html – Lippi

Answer 5

Azul's CTO, Gil Tene, has a nice overview of the problems associated with Garbage Collection and a review of various solutions in his Understanding Java Garbage Collection and What You Can Do about It presentation, and there's additional detail in this article: http://www.infoq.com/articles/azul_gc_in_detail.

Azul's C4 Garbage Collector in our Zing JVM is both parallel and concurrent, and uses the same GC mechanism for both the new and old generations, working concurrently and compacting in both cases. Most importantly, C4 has no stop-the-world fall back. All compaction is performed concurrently with the running application. We have customers running very large (hundreds of GBytes) with worse case GC pause times of <10 msec, and depending on the application often times less than 1-2 msec.

The problem with CMS and G1 is that at some point Java heap memory must be compacted, and both of those garbage collectors stop-the-world/STW (i.e. pause the application) to perform compaction. So while CMS and G1 can push out STW pauses, they don't eliminate them. Azul's C4, however, does completely eliminate STW pauses and that's why Zing has such low GC pauses even for gigantic heap sizes.

And to correct a statement made in an earlier answer, Zing does not require any changes to the Operating System. It runs just like any other JVM on unmodified Linux distros.

Wasp

I just wonder how Azul's C4 achieved what you said and why Sun or Oracle cannot. Is there some big secret or this is just kind of some trade-off? – Humboldt

Azul's C4 has very unique technology that has its origins in Azul's hardware compute appliances (which uses specialized processors built to run enterprise Java apps) and has been evolved to run on regular x86 servers running Linux. Every other enterprise-class garbage collector (whether from Oracle or IBM) at some point must do stop-the-world pauses -- the unique attribute of Azul's C4 is that it never does these problematic STW pauses. If you're curious, the inventors of the C4 collector published a paper on how it works: dl.acm.org/citation.cfm?id=1064988. – Wasp

Scott, I read here blog.mikemccandless.com/2012/07/… that Azul ships a kernel module that pre-allocates memory for JVM usage. Is this not true? If true, not much of a kernel modification but still a modification. – Murder

George, two words: patent protected. Dan, when you purchase Zing, part of what you are paying for is having their support people tune it for your application-- and that includes allocating overall system memory usage. The kernel module itself prevents writes to a memory block that is being garbage collected. That's the secret sauce that makes it "pauseless": threads only pause if they try to write to one of those blocks, and then only long enough to compact that block. – Pool

Answer 6

The G1 collector reduces the impact of full collections. If you have an application where you have already reduced the need for full collections, the Concurrent map Sweep collector is just as good and in my experience has shorter minor collection times.

Omaromara

"note that production use of G1 is only permitted where a Java support contract has been purchased.", groups.google.com/forum/#!topic/javaposse/Vm0a4H-QY54, so it is a myth or not ? – Arlinearlington

@ChristopheRoussy I don't know if this is true any more (or indeed have proof it was ever true) It doesn't require the -XX:+UnlockCommercialFeatures so I suspect that G1 doesn't require a license. – Omaromara

Answer 7

Recently I have been moved from

CMS to G1GC with 4G heap & 8 core processor on servers with JDK 1.7.45.

(JDK 1.8.x G1GC is preferred over 1.7 but due to some limitations, I have to stick to 1.7.45 version)

I have configured below key parameters and kept all other parameters to default values.

-XX:G1HeapRegionSize=n, XX:MaxGCPauseMillis=m, -XX:ParallelGCThreads=n, 
-XX:ConcGCThreads=n apart from -Xms and -Xmx

If you want to fine tune these parameters, have a look at this oracle article.

Key observations:

1. Memory usage is consistent with G1GC unlike high & lows with CMS
2. Max GC pause time is less compared to CMS
3. Time spent in Garbage collection is little bit high in G1GC compared to CMS.
4. Number of major collections are almost negligible compared to CMS
5. Number of minor collections are on higher end compared to CMS

But still I am happy that Max GC pause time is less than as of CMS. I have set Max GC pause time as 1.5 seconds and this value has not been crossed yet.

Related SE question:

Java 7 (JDK 7) garbage collection and documentation on G1

Lauretta

Answer 8

It seems like G1 starting JDK7u4 is finally officially supported, see the RN for JDK7u4 http://www.oracle.com/technetwork/java/javase/7u4-relnotes-1575007.html.

From our testing still for big JVMs tuned CMS still acts better than G1 but I guess it will grow better.

Supplemental

Answer 9

CMS can lead to slowly degraded performance even if you are running it without accumulating tenured objects. This is because of memory fragmentation which G1 supposedly avoids.

The myth about G1 available only with paid support is just that, a myth. Sun and now Oracle have clarified this on the JDK page.

Brittaney

Answer 10

G1 GC is supposed to work better. But if setting -XX:MaxGCPauseMillis too aggressively, garbage will be collecting too slowly. And that's why full GC triggered in David Leppik's example.

Devitrify

Answer 11

I have just implemented G1 Garbage Collector in our Terracotta Big Memory project. While working on different types of collectors G1 gave us the best results with less than 600ms respond time.

You can find the test results (26 in total) here

Hope it helps.

Sabec

Answer 12

I have recently migrated part of Twicsy to a new server with 128GB RAM and decided to use 1.7. I started off using all the same memory settings as I used with 1.6 (I have several instances running doing various things, anywhere from 500mb of heap up to 15GB, and now a new one with 40GB) and that did not work out well at all. 1.7 seems to use more heap than 1.6, and I experienced a lot of issues over the first few days. I luckily had plenty of RAM to work with and bumped up the RAM for most of my processes, but still was having some issues. My normal MO was to use a very small minimum heap size of 16m, even with a max heap of several gigabytes, then turn on incremental GC. This kept the pauses at a minimum. That doesn't work now though, and I had to increase the minimum size to about what I expected to use on average in the heap, and that has worked out very well. I still have incremental GC turned on, but I will be trying it without. No pauses whatsoever now, and things seem to be running very fast. So, I think the moral of the story is don't expect your memory settings to translate perfectly from 1.6 to 1.7.

Syllabus

Answer 13

G1 makes the application a lot more agile: the latancy of the application will raise - the app can be named as "soft-real-time". This is done by replacing two kinds of GC runs (small minor ones and one big on Tenured Gen) to equal-sized small ones.

For more details look at this: http://geekroom.de/java/java-expertise-g1-fur-java-7/

Precursory

Answer 14

I'm working with Java, for small and large Heap, and the question of the GC and Full GC appears every day, as the constraints may be more strict than others : in certain environment, 0.1 second of scavenger GC or Full GC, kill simply the fonctionnalité, and have fine grained configuration and capability is important (CMS, iCMS, others ... the target is here to have the best possible response time with the nearly real time treatment (here the real time treatment is often 25 ms ), so, basically, any improvements in GC ergonomy ans heuristique are welcome !

Anent

Answer 15

I use G1GC on Java 8 and also with Groovy (also Java 8), and I am doing various kinds of workloads, and overally G1GC works like this:

• The memory usage is very low, e.g. 100MB instead of 500MB compared to default Java settings

• The response time is consistent and very low

• The performance between default settings and G1GC is 20% slowdown when using G1GC in worst case scenario (without tuning, single-threaded application). It's not much considering good response time and low memory usage.

• When running from Tomcat which is multi-threaded, the overall performance is 30% better and memory usage is much lower as well response times are much lower.

So in overall, when using really various workloads, G1GC is very good collector for Java 8 for multi-threaded applications, and even for single-threaded there are some benefits.

Club

Answer 16

It's not suggested to use java8 w/ G1GC for float point calculation with hotspot-like JVM. It's dangerous for application integrity & accuracy.

https://bugs.openjdk.java.net/browse/JDK-8148175

JDK-8165766

JDK-8186112

Aileen