Android FinalizerDaemon hanging up

Question

I'm having a really weird problem in an Android app. After a certain point (around when the main activity starts and a fragment is displayed) the FinalizerDaemon just stops processing objects and garbage keeps piling up. Looking at a thread dump, it seems to be stuck on ReferenceQueue.remove() :

"FinalizerDaemon@4461" daemon prio=5 waiting
  java.lang.Thread.State: WAITING
      at java.lang.Object.wait(Object.java:-1)
      at java.lang.Object.wait(Object.java:423)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:101)
      - locked <0x1173> (a java.lang.ref.ReferenceQueue)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:72)
      at java.lang.Daemons$FinalizerDaemon.run(Daemons.java:185)
      at java.lang.Thread.run(Thread.java:818)

Yet the queue is not empty. If I dump the heap after using app for a while, the queue is literally thousands of entries long. The data structure also doesn't look broken:

Dumping again after allocating and garbage collecting some more shows that the head of the queue is the same Matrix instance as before.

Now, I noticed this because I'm retaining some C++ objects, which need to be released at some point. While I suspect that a finalizer calling into JNI functions and doing something stupid on the C++ side could somehow break it, all my logs indicate that all the finalizers are running fine and returning without throwing anything until they just randomly stop getting called. Also, it shouldn't really be possible for a finalize call to break the Daemon, short of segfaulting the whole app or something, since the Watchdog is supposed to handle finalizers that run for too long and throw an exception.

I tried an explicit System.runFinalization() and all it does is hang the main thread forever, waiting for the daemon which never runs.

Any idea how this could happen?

Checklist

Looking at android.googlesource.com/platform/libcore/+/…, it's waiting forever for something to remove. Assuming it's stuck here (vs. running very quickly and you just happen to catch it here), that would suggest that new references are not being added. If you see head != null on that queue, then it would appear that whatever the GC is doing to add references is failing to signal the queue. What version of Android? – Banner

@Banner I edited the question. Taking repeated heap dumps shows that the head of the queue is not null and it doesn't move from the same object. None of my logs in finalizers print either after the point when it seems to get stuck. It would be pretty unlikely that I just randomly catch it on wait every time I try, in different parts of the app and on different devices. I've reproduced it on a Nexus 4 (5.1) and an x86 6.0 emulator, and I have reason to suspect that it happens on a dozen other test devices which I haven't gotten around to connecting to my Android Studio yet. – Checklist

It certainly sounds like the ReferenceQueue isn't behaving correctly. Either this is a widespread problem that nobody has happened to notice (unlikely), or your app has inadvertently mucked things up. Maybe some native code wrote outside the object boundaries and trashed a monitor lock, so notify() no longer works? You could try attaching a debugger and manually advancing or clearing head to see if it starts working when you drop the current entry (and then create some more garbage to cause the queue to signal). – Banner

I reported an issue here: code.google.com/p/android/issues/detail?id=215906 – Okinawa

Answer 1

I believe this has to do with some objects being resurrected in their finalize methods.

I'll quote a paragraph from this question.

The finalizer thread runs so the garbage collection operates to clean up resources associated with an object. If I'm seeing it corectly, the finalizer can't get the lock to this object: java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) because the java object is running a method, so the finalizer thread is locked until that object is finished with it's current task.

Maybe that's the situation you have.

Madeup

I'm afraid this is not the case. What the Threaddump says is just the FinalizerDaemon is waiting forever for an object to become available on the ReferenceQueue. But why are there no objects coming in? – Okinawa

If you are right and I'm overlooking something: Can you tell exactly which java object is running a method that locks the finalizer? – Okinawa

Here is a Threadump: code.google.com/p/android/issues/… – Okinawa

I'm not sure but I see that the WAITING threads have to do with some ThreadPoolExecutor, if you use any make sure that you invoke shutdown() method when the wrapping Activity/Fragment is stopped. Also try enabling StrictMode, it may show more info developer.android.com/reference/android/os/StrictMode.html – Madeup

The queue should be cleared while the Activity is running. Also I don't use any ThreadPoolExecutor. Those that are implicitely used by Fresco are shutdown by the library itself. – Okinawa

I suggest you post your Fragment Code, or try commenting out the Fresco loading methods and see what happens, maybe this can tell us which module is causing the problem, then we can troubleshoot it. – Madeup

The Fresco loading is used by react-native. This happens in the sample project of react-native, without any other code. – Okinawa