Intermittent error org.hibernate.PersistentObjectException: detached entity passed to persist

Question

I am getting the exception org.hibernate.PersistentObjectException: detached entity passed to persist. From the numerous posts on this forum and elsewhere, I understand that this happens in two cases(not considering One-One annotations etc),

1. There is an issue with the transaction going out of scope
2. An id is set where it should be automatically generated.

I see neither of these happening with my code. I am unable to reproduce the error, because I don't have the data which initially triggered it. On other data it runs perfectly fine. I have provided an SCCE below:

public class MyProcessor {
    private MyImportEJB myEJB = MyImportEJB.getInstance();
    private List<MyClass> saveQueue = new ArrayList<MyClass>();
    public void process() {
        List<X> rawData = getListOfX();
        for(X x:rawData) {
             processX();
         }
         saveFoos(saveQueue);   
     }

     public void saveOrUpdateFoos(List<Foo> foos) {

        for(MyClass foo:foos) {
              MyClass existingFoo = myEJB.getFoosForWidAndDates(foo.getWid(), foo.getEffBeginDt(),foo.getEffEndDt());
              if(existingFoo == null) saveQueue.add(foo);
              else {
                   existingFoo.updateIfDifferent(foo);
                   saveQueue.add(existingFoo);
              }
         }

         if(saveQueue.size() > 5000) {
             myEJB.saveObjects(saveQueue);
             saveQueue.clear();
         }
     }

     public void processX() {
          ArrayList<MyClass> foos = new ArrayList<MyClass>();

          if(x.reportPeriod != null && x.gravity != null){
              MyClass foo = new MyClass();
              foo.setId(null);
              foo.setWId(x.getWid());
              foo.setEffBeginDt(x.reportPeriod);
              foo.setEffEndDt(addOneMonth(x.reportPeriod));
              foo.setGravity(x.gravity);

              foos.add(foo);
          }
          saveOrUpdateFoos(foos);

     }
 }

MyImportEJB.java:

@Stateless
@EJB(name = "MyImportEJB", beanInterface = MyImportEJB.class)
@TransactionAttribute(TransactionAttributeType.SUPPORTS)
@PermitAll
public class MyImportEJB{
    @Override
    @TransactionAttribute(TransactionAttributeType.REQUIRED)
    public void saveObjects(List<? extends P> mappedObjects) 
    {
        for (P mappedObject : mappedObjects)
        {
            this.saveObject(mappedObject);
        }
    }


    @Override
    @TransactionAttribute(TransactionAttributeType.REQUIRED)
    public void saveObject(P mappedObject) 
    {
        EntityManager entityManager = this.getEntityManager();

        Object identifier = this.getEntityManagerFactory().getPersistenceUnitUtil().getIdentifier(mappedObject);
        if (identifier != null) {
            Object existingObject = entityManager.find(mappedObject.getClass(), identifier);

            if (existingObject != null) {
                entityManager.merge(mappedObject);
                return;
            }
         }

         entityManager.persist(mappedObject);
    }

    public MyClass getFoosForWidAndDates(Integer wid, Calendar effBeginDt, Calendar effEndDt) {
         try {
            return (MyClass)((this.entityManager
        .createQuery("select M from MyClass M where wid = :wid and effBeginDt = :effBeginDt and effEndDt = :effEndDt ", MyClass.class)
        .setParameter("wid",wid)
        .setParameter("effBeginDt", effBeginDt)
        .setParameter("effEndDt", effEndDt)).getSingleResult());
         } catch(NoResultException | NonUniqueResultException e) {
            return null;
        }
    }
 }

MyClass.java

public MyClass{

      @Id
      @GeneratedValue(strategy=GenerationType.IDENTITY)
      @Column(name = "Id")
      private Integer id;

      @Column(name = "wid")
      private Integer wId;

      @Column(name = "eff_begin_dt")
      private Calendar effBeginDt;

      @Column(name = "eff_end_dt")
      private Calendar effEndDt;

      @Column(name = "gravity")
      private Double gravity;

      private Integer dataDownloadId;

      public void updateIfDifferent(MyClass other) {
          if(other.gravity != null && other.gravity != this.gravity) this.gravity = other.gravity;
          //same for effBeginDt andeffEndDt

      }

 }

persistence.xml

  <?xml version="1.0" encoding="UTF-8" ?>

 <persistence xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
version="2.0">
    <persistence-unit name="ProdData">
        <description>ProdData Database Persistence Unit</description>
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <jta-data-source>java:jboss/ProdDataJNDI</jta-data-source>
        <class>path.to.MyClass</class>
        <class>path.to.X</class>
        <exclude-unlisted-classes>true</exclude-unlisted-classes>

        <properties>
            <property name="hibernate.show_sql" value="false" />
         </properties>
    </persistence-unit>
 </persistence>

The exception is thrown on calling entityManager.persist(mappedObject) <- MyImportEJB.saveObject <-MyImportEJB.saveObjects. I dont have the line number

I have tried writing a sample program where I get an existingFoo object from the database, update and save it, because that was the most likely source of the error. But I could not reproduce the error. Please help.

EDIT: Here are the details of getListofX() as requested

from MyProcessor.java:

public List<X> getListOfX() {
    return myImportEJB.getUnprocessedIds(X.class, 30495);
}

from the file MyImportEJB.java:

@TransactionAttribute(TransactionAttributeType.SUPPORTS)
public List<Integer> getUnprocessedIds(Class<? extends ProductionRawData> clazz, Integer dataDownloadId) {
    String canonicalName = clazz.getCanonicalName();

    String queryStr = "select id from " + canonicalName + " where datadownloadId = :dataDownloadId and isProcessed != 1";

    TypedQuery<Integer> query = this.entityManager.createQuery(queryStr, Integer.class)
        .setParameter("dataDownloadId", dataDownloadId);
    try {
        return query.getResultList();
    } catch(NoResultException nre) {
         return new ArrayList<T>();
    }
}

EDIT: Also added the details of getFoosForWidAndDates(). It was suggested to me that I set the id on a new Foo to null before adding it to the save queue. I would like to know if it is possible that the id is being set "under the hood" by Hibernate to an unacceptable value

Adulterate

What is the source of getListOfX() ? Does your entity has any relationship with another entity ? I guess this question is a duplicate, this could help you. Otherwise you have that other exchange, but it doesn't fit your situation. – Lombroso

The entity I'm saving has no relationship with any other entity – Adulterate

How does MyImportEJB.getInstance() work? If the MyImportEJB is not created by the container then none of the transaction management semantics implied by your annotations will be having any effect. The fact that you do not appear to be injecting an EntityManager with @PersistenceContext makes me a bit suspicious because it suggests that injection was not working for you. – Hoffert

It is injected using @PersistenceContext, I omitted that for brevity – Adulterate

Answer 1

I ran the code on the production server again and I got a TransactionRollbackException right before the detached entity exceptions. I think that this might be the actual cause of the exception I observed.

The transaction timed out and was rolled back due to the huge number of objects I was trying to save, and this led to the Foo objects becoming detached. Somehow the TransactionRollbackException didnt show up in the logs the first time around, probably because it happened exactly at 12:0am UT. In support of this theory, the program hasn't crashed so far after I decreased the batch size to ~1000

For those interested, a TransactionRollbackException does lead to entities getting detached: entityManager.getTransaction().rollback() detaches entities?

There are ways to handle this when it occurs, but I opted against implementing them for simplicity's sake

Adulterate

Answer 2

I think this could be your issue

 if(saveQueue.size() > 5000) {
     myEJB.saveObjects(saveQueue);
     saveQueue.clear();
 }

You are filling the saveQueue up to 5000, and then you try to persist / merge them. I'm not sure about Hibernate, but EclipseLink's default shared cache size is 1000. That means, you could fill up saveQueue with, for example, 2000 existing Foos and, by the time you call persist() on them, they are no longer in entity manager's cache. This leads to calling persist() on detached instance which has id already set.

In your case, I think it would be perfectly OK to call entityManager.merge(mappedObject) on all objects, regardless of weather it is an existing object or not, because it will update existing ones, and persist new ones.

Ity

How can I be sure that this is the issue? Is there a way to, say, decrease the cache size and make it trigger this kind of behavior? I need to be sure I've fixed the code before running it in production – Adulterate

I've only dealt with EclipseLink's cache settings in the past, so either Hibernate is doing it in a different way, or it has changed a lot since the time I used it. I can't find a way to set default L2 cache size, I can't even find a confirmation that it is enabled by default. But, if you can write a test with 5000+ existing Foos (if there is a default cache size, I'm sure it's not > 5000) you could test this theory. If the test works, then I'm wrong. – Ity

Also, I'm not sure why it would call persist()? In the function saveOrUpdate, it is calling entityManager.find() to check if the object already exists in the database. If it exists it calls merge(). Based on this answer, it seems that find doesn't look for the object in the cache but actually queries the database for the object – Adulterate

merge() has a more complex lifecycle than persist(), but it does save an entity if it is new, check this for details. However, I'm generally against using merge() on new objects, but in your case I think it would make sense to use it, of course only if my assumption about cache is right. If not, I wouldn't recommend it. – Ity

I wrote a test with >5000 existing Foos and it didn't throw that exception – Adulterate

Then I'm wrong, but it was an interesting theory. On closer look, if yog get exception at persist(), then it is a problem with new objects, not existing. Hard to tell without reproducable test case. Maybe the code of getFoosForWidAndDates() would help – Ity

Upvoting your answer simply because decreasing the batch size did apparently fix the problem – Adulterate

Answer 3

I hope you are using latest Hibernate. And I expect that it is a typo here:

if(existingFoo == null) saveQueue.add(existingFoo); // if it is a null, you add it?

Otherwise it looks quite interesting, you could try to do following things:

1. Flush session after each (10, 1000) persisted objects, like that:

   public void saveObjects(List<? extends P> mappedObjects) 
   {
       for (P mappedObject : mappedObjects)
       {
           this.saveObject(mappedObject);
           // some condition if you have a lot of objects
           this.entityManager.flush();
       }
   }
   

2. Use less exotic way to check if entity was persisted already:

   public void saveObject(P mappedObject) {
       EntityManager entityManager = this.getEntityManager();
   
       if (mappedObject.getId() != null) {
           entityManager.merge(mappedObject);
           return;
        }
   
        entityManager.persist(mappedObject);
   }
   

   Or it is better completely avoid merge and just work with managed objects, this will require a single transaction for all operations, thus will require redesign of the things.

Lethe

I fixed the typo with the existingFoo, it should actually save the foo object. So I could try your suggestions, but the thing is that I want to identify the root cause of the problem before running it in a production environment. For some reason, the exception that is getting thrown is filling up the logs even though I have it in a catch block. This is crashing the server due to disk space constraints. I need either a solution that identifies the problem correctly, or prevents it altogether(say like your managed objects) – Adulterate

Answer 4

I had a similar issue within my spring application in a method with transactional annotation. For ex @Transactional public void method1(){ //line1 //line2 //line3 } In the above code, if an exception happened at line2 then the transaction is rolled back and entities at line3 will be detached and thrown as detached entity. I think we need to wrap the code in try .. catch and log the exception properly then threw a custom exception.

Sellma