Why GC doesn't collect my objects?

Question

I have a java program which keeps calling java.util.zip to compress/decompress data. It runs out of memory within a few seconds. I had a memory dump with jmap and I'm viewing it with jhat.

Finalizer summary shows Total instances pending finalization: 0. If I understand it correctly, I don't have any objects which (1) have finalize() method, (2) have been marked by GC and (3) are waiting to be finalized. This seems good.

When I look at a particular object, the only reference to this object is a java.lang.ref.Finalizer. The Finalizer object is created for each object which has a finalize() method no matter if the object is GC'ed or not. So it looks like nothing is preventing this this Deflater object to be GC'ed.

Object at 0x7f4aeb7a35d0

instance of java.util.zip.Deflater@0x7f4aeb7a35d0 (51 bytes)

References to this object:

java.lang.ref.Finalizer@0x7f4aeb8607c8 (64 bytes) : field referent

The program is paused in the run by System.in.read(). The memory usage doesn't go down after a while.

UPDATE:

I should make it clear. The memory dump shows many objects were not GC'ed but no other objects (except Finalizer objects) were referencing them. I'm trying to find out why they were not GC'ed.

Renell

@dashrb I don't work with Deflater in my code. I'm using Apache Thrift which uses Deflater. Deflater is not in scope as Show all members of the rootset shows only 1 reference to Deflater class but to no instances. – Renell

Actually, the fact that you are running out of memory and have no objects that have been marked by GC is bad. That means your code (or some library code you use) is keeping references to objects so the GC can't collect them. – Monumental

@TedHopp that's what I'm going to find out. I don't know how to find those references as memory dump shows there were not any references. – Renell

If you use jmap to print a histogram of object allocations, that might show what kind of objects are chewing up memory and give you a clue as to where to look next. There's a nice article here on tools for tracking down memory leaks. – Monumental

Why not use a memory profile to see what exactly is alive and who is referencing whom? – Heathen

You could try something more user-friendly to understand your memory dump... Eclipse MAT is an option: eclipse.org/mat – Gracia

OutOfMemory error is usually contains a hint whether there is a problem with heap, PermGen, fragmentation, GC, etc. Are you sure your heap is exhausted? – Funda

Make a for loop that will call System.gc() 5 times. Report back. – Bigner

Unzip the file manually and take a look at how much data is inside. All of the Java Zip implementations I have found have made the assumption that the entire file can be read into memory and decompressed there. – Hagfish

Answer 1

Are you positive you closed the stream and invoked end on the deflater? I'm sorry if this is a simplistic suggestion you've already tried, but there were a lot of complaints about a memory leak when using Deflater when not immediately invoking end on it, for example:

• http://www-01.ibm.com/support/docview.wss?uid=swg21227106
• https://bugs.java.com/bugdatabase/view_bug?bug_id=6734186 (1.5)
• excessive memory use by java
• https://bugs.java.com/bugdatabase/view_bug?bug_id=4797189
• https://bugs.java.com/bugdatabase/view_bug?bug_id=6293787
• http://kohsuke.org/2011/11/04/quiz-answer-memory-leak-in-java/

The root cause is apparently that the collector is not able to keep up with the application when memory used by native elements is involved. That also explains the behavior you are seeing when profiling: the memory is ready to be reclaimed, but just isn't reclaimed quickly enough.

Since you wrote you don't use a Deflator directly but through Apache Thrift, try to identify which method in that library is responsible for ending the deflator, and made sure you invoked that method.

Hodess

Answer 2

The base problem is that you aren't freeing objects from the C heap. Many classes in java.util.zip use Deflater. Deflater maintains a reference to data in the C heap. Your code is likely not calling close() on ZipOutputStream or DeflaterOutputStream, or it's not calling end() on Deflater.

(If you are passing your own Deflater to ZipOutputStream or DeflaterOutputStream, you are responsible for calling end() on the Deflater.)

GC is of limited help in your case because the stream(s) must be dereferenced and eventually finalized. This may take multiple GC sweeps. I experienced a similar problem with Jetty and proposed a fix for it.

Goar

Answer 3

Adding to Marian-Daniel Craciunescu's answer

I did have this kind of problem with another type of use.

The solution I found is the following :

• Put the object into a map or somewhere you can easily drop its reference
• Put also the object into a WeakReference for using it
• Use your reference as normal
• Delete the reference from the map where no more used

Be careful of inherited references

It is greatly possible your object is "hidden-inner-referenced" in another object, use the debugger and "expand" all objects using it.

You will never know when the GC is running, nor what it will handle (except objects in use).

Useful link

You should read (if it is not already done) this post about WeakReferences : Understanding Java's Reference classes: SoftReference, WeakReference, and PhantomReference

Conglomerate

Answer 4

Firstly before thinking about memory leak we should try to understand if the memory requirement is normal for the processing you are doing. What is the size of memory available to JVM i.e -XmX. If you increase the size of memory or decrease the size of file being processed does work?

Chenee

Answer 5

Maybe you should use a WeakReference for your object.In this way GC may finalize () your object much quickly

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/ref/WeakReference.html http://weblogs.java.net/blog/2006/05/04/understanding-weak-references

Dudleyduds