Out of Memory When Loading Java Entities

Question

I have a terrible problem that hopefully has a very simple answer. I am running out of memory when I perform a basic

If I have code like this:

MyEntity myEntity;
for (Object id: someIdList) {
   myEntity = find(id); 
   // do something basic with myEntity
}

And the find() method is a standard EntityManager related method:

public MyEntity find(Object id) {
    return em.find(mycorp.ejb.entity.MyEntity.class, id);
}

This code worked a couple of weeks ago, and works fine if there are fewer items in the database. The resulting error I am facing is:

java.lang.OutOfMemoryError: GC overhead limit exceeded

The exception is coming from oracle toplink calling some oracle jdbc methods.

The loop exists because an EJBQL such as "select object(o) from MyEntity as o" will overload the application server when there are lots of records.

Polard

Performing the em.clear() isn't the answer I am looking for... but I will keep it in mind. The issue with this problem is that it used to work... with the same level of database entries! And this raises another question - If I have a loop, I can flush/clear periodically.. but what about when the application is simply running for a long period of time? The EntityManager should manage how much it's managing! – Polard

The EntityManager has most of time a short lifecycle (the common pattern is entitymanager-per-request). I suggest reading 5.1. Entity manager and transaction scopes. If you don't follow this pattern, then you'll have to deal with "memory handling". The EntityManager keeps entities in memory to track changes, that's how JPA works, you can't reasonably expect the EntityManager to be able to load a whole database or to flush changes and detach entities for you. – Concourse

Answer 1

This code worked a couple of weeks ago, and works fine if there are fewer items in the database. The resulting error I am facing is: java.lang.OutOfMemoryError: GC overhead limit exceeded

And there is nothing surprising here. Entities loaded by em.find() are put and kept in the persistence context (in memory) to track changes so if you bulk load too much of them without precautions, you'll just explode your memory and get OOME.

If you really need to do something with all of your entities, you need to call flush() first to push all the changes to the database and then clear() to clear the persistent context and release memory at regular intervals:

int i = 0;
for (Object id: someReallyBigIdList) {
    myEntity = find(id); 
    // do something basic with myEntity
    if ( i % 20 == 0 ) { //20, same as the JDBC batch size
        //flush a batch of DML operations and release memory:
        em.flush();
        em.clear();
    }
    i++;
}

Calling clear() causes all managed entities to become detached. Changes made to entities that have not been flushed to the database will not be persisted. Hence the need to flush() the changes first.

Concourse

Answer 2

The problem is, if you do the loop you just query one entity after another, but it is still referenced in your EntityManager. You must either

• clear() the entityManager, or
• remove the entity from it (forgot how the function for this was called).

It is also a good idea to set the entity manager readonly if possible, because that stops JPA from holding a copy of each of your objects in memory, just to detect if it has possible changes in case of a flush() to the database.

Marciano

I just encounter an issue in Hibernate, putting the entity as read-only doesn't mean it stop persistent context to hold a copy in persistent context, it will still hold, because subsequent load/find will fetch a copy from persistent context instead of database. – Zwickau