Can the OS stop a Java process from garbage collecting?

Question

I'm monitoring a production system with AppDynamics and we just had the system slow to a crawl and almost freeze up. Just prior to this event, AppDynamics is showing all GC activity (minor and major alike) flatline for several minutes...and then come back to life.

Even during periods of ultra low load on the system, we still see our JVMs doing some GC activity. We've never had it totally flatline and drop to 0.

Also - the network I/O flatlined at the same instance of time as the GC/memory flatline.

So I ask: can something at the system level cause a JVM to freeze, or cause its garbage collection to hang/freeze? This is on a CentOS machine.

Pierpont

"the system slow to a crawl and almost freeze up" is there a memory leak? – Dogtired

Yeah I know it sounds that way, but we'd see the amount of available memory shrink over time, which we don't. Definitely a good thought though. – Pierpont

I've seen something like you describe before because Statement objects were never closed and, after a while, filled the heap. Use VisualVM or whatever tool you like to check what uses most of the memory and see if it keeps growing. – Dogtired

We've done that very same memory leak analysis on non-production servers and haven't found anything. I really dont think this is a memory leak... – Pierpont

Answer 1

Does your OS have swapping enabled.

I've noticed HUGE problems with Java once it fills up all the ram on an OS with swapping enabled--it will actually devistate windows systems, effictevly locking them up and causing a reboot.

My theory is this:

• The OS ram gets near full.
• The OS requests memory back from Java.
• This Triggers Java into a full GC to attempt to release memory.
• The full GC touches nearly every piece of the VMs memory, even items that have been swapped out.
• The system tries to swap data back into memory for the VM (on a system that is already out of ram)
• This keeps snowballing.

At first it doesn't effect the system much, but if you try to launch an app that wants a bunch of memory it can take a really long time, and your system just keeps degrading.

Multiple large VMs can make this worse, I run 3 or 4 huge ones and my system now starts to sieze when I get over 60-70% RAM usage.

This is conjecture but it describes the behavior I've seen after days of testing.

The effect is that all the swapping seems to "Prevent" gc. More accurately the OS is spending most of the GC time swapping which makes it look like it's hanging doing nothing during GC.

A fix--set -Xmx to a lower value, drop it until you allow enough room to avoid swapping. This has always fixed my problem, if it doesn't fix yours then I'm wrong about the cause of your problem :)

Tuneless

How exactly can a OS ask for memory back from a process? – Bouilli

I believe the OS can signal a low memory condition, but I can't find a reference. I may be wrong but the effect is as described--the system grinds to a halt after it runs out of ram regardless of how much swap is available if a JAVA app is taking up most of the ram. I've seen this in Linux but it is much more pronounced on windows. When I had my -Xmx set wrong I've had to reboot my machine when it hit the ram limit. Also GC seems to stop during this time because the OS is spending all it's time swapping (Which is as TicketMonster described.) – Tuneless

What about this signal? msdn.microsoft.com/en-us/library/windows/desktop/… If Java responds to this, it would explain the behavior as I've described. – Tuneless

Answer 2

It is really difficult to find the exact cause of your problem without more information.

But I can try to answer to your question :
Can the OS block the garbage collection ?
It is very unlikely than your OS blocks the thread garbage collector and let the other threads run. You should not investigate that way.

Can the OS block the JVM ?
Yes it perflecty can and it do it a lot, but so fast than you think that the processes are all running at the same time.
jvm is a process like the other and his under the control of the OS. You have to check the cpu used by the application when it hangs (with monitoring on the server not in the jvm). If it is very low then I see 2 causes (but there are more) :

• Your server doesn't have enough RAM and is swapping (RAM <-> disk), process becomes extremely slow. In this case cpu will be high on the server but low for the jvm
  
• Another process or server grabs the resources and your application or server receive nothing. Check the priority on CentOs.

Parasite

Answer 3

In theory, YES, it can. But it practice, it never should.

In most Java virtual machines, application threads are not the only threads that are running. Apart from the application threads, there are compilation threads, finalizer threads, garbage collection threads, and some more. Scheduling decisions for allocating CPU cores to these threads and other threads from other programs running on the machine are based on many parameters (thread priorities, their last execution time, etc), which try be fair to all threads. So, in practice no thread in the system should be waiting for CPU allocation for an unreasonably long time and the operating system should not block any thread for an unlimited amount of time.

There is minimal activity that the garbage collection threads (and other VM threads) need to do. They need to check periodically to see if a garbage collection is needed. Even if the application threads are all suspended, there could be other VM threads, such as the JIT compiler thread or the finalizer thread, that do work and ,hence, allocate objects and trigger garbage collection. This is particularly true for meta-circular JVM that implement VM threads in Java and not in a C/C++;

Moreover, most modern JVM use a generational garbage collector (A garbage collector that partitions the heap into separate spaces and puts objects with different ages in different parts of the heap) This means as objects get older and older, they need to be moved to other older spaces. Hence, even if there is no need to collect objects, a generational garbage collector may move objects from one space to another.

Of course the details of each garbage collector in different from JVM to JVM. To put more salt on the injury, some JVMs support more than one type of garbage collector. But seeing a minimal garbage collection activity in an idle application is no surprise.

Acerate