Quartz JobStore: JobPersistenceException: Couldn't retrieve trigger: ClassNotFoundException

Question

I'm hoping one of you lovely people will be able to help me with this, as I've spent a number of fruitless hours already trying to make everything play nice!

I've traced the issue down to Classloading, and been able to see that when Quartz tries to de-serialise jobDetail's from a jobStore (jobStoreCMT), the Classloader used does not contain any of my applications classes, and only the libraries defined in the EARs lib folder.

So... I'm obviously using an application server, and in this case tried against Glassfish 3.1.1/3.1.2

tried against Quartz 1.8.6/2.1.5 using Spring 3.1.0.RELEASE

Spring/Quartz config:

<bean id="schedulerFactoryBean" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="overwriteExistingJobs" value="true" />

    <property name="triggers">
        <list>
            <ref bean="notificationEmailsSimpleTrigger" />
        </list>
    </property>

    <property name="quartzProperties">
        <props>
            <prop key="org.quartz.scheduler.instanceName">QuartzScheduler</prop>
            <prop key="org.quartz.scheduler.instanceId">AUTO</prop>

            <prop key="org.quartz.threadPool.class">org.quartz.simpl.SimpleThreadPool</prop>
            <prop key="org.quartz.threadPool.threadCount">25</prop>
            <prop key="org.quartz.threadPool.threadPriority">5</prop>
            <prop key="org.quartz.jobStore.class">org.quartz.impl.jdbcjobstore.JobStoreCMT</prop>
            <prop key="org.quartz.jobStore.driverDelegateClass">org.quartz.impl.jdbcjobstore.StdJDBCDelegate</prop>
            <prop key="org.quartz.jobStore.misfireThreshold">60000</prop>
            <prop key="org.quartz.jobStore.tablePrefix">QRTZ_</prop>
            <!-- <prop key="org.quartz.jobStore.isClustered">true</prop> -->
            <!-- <prop key="org.quartz.jobStore.clusterCheckinInterval">20000</prop> -->

            <prop key="org.quartz.scheduler.classLoadHelper.class">org.quartz.simpl.CascadingClassLoadHelper</prop>
            <prop key="org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer">true</prop>
            <prop key="org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread">true</prop>

            <prop key="org.quartz.scheduler.skipUpdateCheck">true</prop>
        </props>
    </property>
</bean>

and the corresponding trigger reference:

<bean id="notificationEmailsSimpleTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerFactoryBean">
    <property name="jobDetail" ref="notificationJobDetail" />
    <property name="repeatInterval" value="60000" />
</bean>

<bean id="notificationJobDetail" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
    <property name="jobClass" value="com.mcboom.social.notifications.NotificationQuartzJobBean" />
</bean>

So the problem I'm having is this: any combination of the below doesn't seem to effect the classloader being used.

    <prop key="org.quartz.scheduler.classLoadHelper.class">org.quartz.simpl.CascadingClassLoadHelper</prop>
    <prop key="org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer">true</prop>
    <prop key="org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread">true</prop>

Or more specifically dont help when trying to retrieve a previously persisted trigger, resulting in the following stracktrace:

INFO: ERROR - ErrorLogger.schedulerError(schedulerFactoryBean_QuartzSchedulerThread)(2358) | An error occured while scanning for the next trigger to fire.
org.quartz.JobPersistenceException: Couldn't acquire next trigger: Couldn't retrieve trigger: com.mcboom.social.notifications.NotificationQuartzJobBean [See nested exception: org.quartz.JobPersistenceException: Couldn't retrieve trigger: com.mcboom.social.notifications.NotificationQuartzJobBean [See nested exception: java.lang.ClassNotFoundException: com.mcboom.social.notifications.NotificationQuartzJobBean]]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2814)
at org.quartz.impl.jdbcjobstore.JobStoreSupport$36.execute(JobStoreSupport.java:2757)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2753)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:263)
Caused by: org.quartz.JobPersistenceException: Couldn't retrieve trigger: com.mcboom.social.notifications.NotificationQuartzJobBean [See nested exception: java.lang.ClassNotFoundException: com.mcboom.social.notifications.NotificationQuartzJobBean]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1596)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.retrieveTrigger(JobStoreSupport.java:1572)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2792)
... 4 more

I can see the org.quartz.simpl.CascadingClassLoadHelper being used on-load, and correctly selecting the right classloader.

Problem is that when the QuartzSchedulerThread tries to retrieve a trigger it uses JobStoreSupport.retrieveTrigger(), which in turn falls-back to ObjectInputsStream.resolveClass(), and the following line of code:

Class.forName(name, false, latestUserDefinedLoader())

Where latestUserDefinedLoader() always returns the wrong classloader...resulting in the ClassNotFoundException and leaving me pretty flummoxed!

I should point out that latestUserDefinedLoader() is a native method of ObjectInputsStream, and i'm using jdk 1.6 on OSX.

Can anyone from either the Quartz/ Spring or more likely Glassfish community shed some light on this, I'm pulling my hair out at the moment.

Thanks Steve.

Sophistication

Answer 1

If anyones interested the solution I came up with, although not ideal was to extend the default StdJDBCDelegate, in my case for MySQL, where I was able to anonymously override ObjectInputStream.resolveClass() much like the below.

/**
 * <p>
 * This method should be overridden by any delegate subclasses that need
 * special handling for BLOBs. The default implementation uses standard JDBC
 * <code>java.sql.Blob</code> operations.
 * </p>
 * 
 * <p>
 * This implementation overcomes the incorrect classloader being used in
 * ObjectInputStream, overriding it with the current threads classloader.
 * </p>
 * 
 * @param rs
 *            the result set, already queued to the correct row
 * @param colName
 *            the column name for the BLOB
 * @return the deserialized Object from the ResultSet BLOB
 * @throws ClassNotFoundException
 *             if a class found during deserialization cannot be found
 * @throws IOException
 *             if deserialization causes an error
 */
@Override
protected Object getObjectFromBlob(ResultSet rs, String colName) throws ClassNotFoundException, IOException, SQLException {
    Object obj = null;

    Blob blobLocator = rs.getBlob(colName);
    if (blobLocator != null && blobLocator.length() != 0) {
        InputStream binaryInput = blobLocator.getBinaryStream();

        if (null != binaryInput) {
            if (binaryInput instanceof ByteArrayInputStream && ((ByteArrayInputStream) binaryInput).available() == 0) {
                // do nothing
            } else {
                ObjectInputStream in = new ObjectInputStream(binaryInput) {

                    @Override
                    protected Class<?> resolveClass(ObjectStreamClass desc) throws IOException, ClassNotFoundException {
                        String name = desc.getName();
                        try {
                            return Class.forName(name, false, Thread.currentThread().getContextClassLoader());
                        } catch (ClassNotFoundException ex) {
                            return super.resolveClass(desc);
                        }
                    }
                };

                try {
                    obj = in.readObject();
                } finally {
                    in.close();
                }
            }
        }

    }
    return obj;
}   

Using the current threads classloader, falling-back to the default handling if unsuccessful.

If anyone has a better solution or even an explanation as to why the issue occurred in the first place, I'd be more than interested in hearing.

S.

Sophistication

A more elegant solution was to repackage my EAR, packaging all non EJB classes in core.jar lib and placing this in the lib folder of the EAR, to share the same classloader as all other application deps. Resolved the issue but carried the unwanted overhead of a significant amount of refactoring. – Sophistication

Answer 2

Use org.springframework.scheduling.quartz.JobDetailBean instead org.springframework.scheduling.quartz.JobDetailFactoryBean

Wargo

org.springframework.scheduling.quartz.JobDetailFactoryBean bean is a drop-in replacement for the org.springframework.scheduling.quartz.JobDetailBean to support Quartx 2.x, so no help there i'm afraid. Thanks for the suggestion tho. – Sophistication

Answer 3

In my case, there was a faulty job that was present in the Quartz database for which no job existed as a bean in the Java configuration, hence it complained that it couldnt find the class. when I deleted the record from the database from 3 tables: qrtz_cron_triggers, qrtz_triggers, qrtz_ob_details, it began to work

Ebonize