JBoss 5.1.0 EAP. Randomly Occuring a java.lang.NoClassDefFoundError

Question

In our EJB 3.0 - application, we have an randomly occuring java.lang.NoClassDefFoundError happening.

It just happens localy in the IDE.

Setup:

OS: Windows 7 32 Bit
Java JDK: 1.6.0_24 (32 Bit)‎
IDE: IntelliJ 11.1‎
App-Server: JBoss 5.1.0 EAP
EAR-Building with „maven-ear-plugin“‎
WAR-Building with Maven „<packaging>war</packaging>“‎
EJB-Building with Maven „<packaging>ejb</packaging>“ und „maven-ejb-‎plugin“‎

EAR looks like this:

xxx.ear
‎   |___ META-INF
‎   |    |___ application.xml (Information about EAR, EJB und WEB-Modul ‎configured)‎
‎   |‎
‎   |___ xxxEJB.jar‎
‎   |    |‎
‎   |    |___ META-INF
‎   |    |    |___ persistance.xml (JPA 1.0 configuration)‎
‎   |    |    |___ jboss.xml (JBoss-Config)‎
‎   |    |    |___ ejb-jar.xml (EJB 3.0 definition)‎
‎   |    |‎
‎   |    |___ country/yyy/xxx/.. (our own compiled classes)
‎   |    |___ report/.. (JasperReport-Tamplates)
‎   |    |___ quickfix/.. (QuickFixJ-configuration)
‎   |    |___ all dependent libraries for our EJB 3.0 project, (74-jar-files)
‎   | ‎
‎   |___ xxxWAR.war‎
‎        |‎
‎        |___ META-INF
‎        |___ WEB-INF
‎             |‎
‎             |___ lib‎
‎             |    |___ all dependent libraries for the Servlet-‎project, (2-jar-files)
‎             |___ classes‎
‎             |    |___ country/yyy/xxx/.. (our own compiled classes)
‎             |___ web.xml (Servlet-configuration)‎

Stacktrace looks like this:

The JVM has been started with the option: -verbose:class to see all the classloading actions.

--- SNIP JBoss-Log ---‎

‎[Loaded country.yyy.xxx.service.issue.util.IssueUtil from file:/C:/Workarea/Projects/xxx-‎trunk/xxx/target/classes/]‎
‎[Loaded org.joda.time.DateTimeZone from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.Chronology from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.chrono.BaseChronology from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.DateTimeZone$1 from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.tz.NameProvider from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.JodaTimePermission from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.tz.Provider from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.tz.FixedDateTimeZone from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
‎[Loaded org.joda.time.tz.ZoneInfoProvider from jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar]‎
ERROR - javax.ejb.EJBTransactionRolledbackException: Unexpected Error
java.lang.NoClassDefFoundError: org/joda/time/tz/DefaultNameProvider
at org.joda.time.DateTimeZone.getDefaultNameProvider(DateTimeZone.java:545)‎
at org.joda.time.DateTimeZone.setNameProvider0(DateTimeZone.java:515)‎
at org.joda.time.DateTimeZone.<clinit>(DateTimeZone.java:116)‎
at ‎country.yyy.xxx.service.issue.util.InitialFixingDateTimeUtil.getHedgingTimestampsForIssue(InitialFixingDateTimeUtil.java:83)‎
at ‎country.yyy.xxx.service.issue.util.InitialFixingDateTimeUtil.getFirstHedgingTimestampForIssue(InitialFixingDateTimeUtil.java:43)‎
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)‎
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)‎
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)‎
at java.lang.reflect.Method.invoke(Unknown Source)‎

‎--- SNAP JBoss-Log ---‎

The file "org/joda/time/tz/DefaultNameProvider" is definitely in the jar file jar:file:/C:/Workarea/Projects/xxx-trunk/xxx-‎ear/target/xxx.ear!/xxxEJB.jar

Suggestions:

We have the feeling that the Quarz has something to do with it. So when something has been scheduled, out of an EJB 3.0 Remote-Stateless-Service. Sometimes the error occures.

The Quartz-Scheduler is initially created in the Servlet-Context, but mostly it works fine.

Did somebody had the same issue?

JBoss Deployment is done via IntelliJ JBoss plugin:

Can it be, that the problem is because of the deployment mechanism we are using? we deploy the application with intelliJ JBoss Plugin, so JBoss says:

09:39:21,614 INFO  [MainDeployer] deploy, url=file:/C:/Workarea/Projects/xxx-trunk/xxx-ear/target/xxx.ear
09:39:33,613 INFO  [Ejb3DependenciesDeployer] Encountered deployment AbstractVFSDeploymentContext@2726974{vfszip:/C:/Workarea/Projects/xxx-trunk/xxx-ear/target/xxx.ear/xxxEJB.jar/}

Juvenility

looks like you trying to load xxxEJB.jar with xxxWAR.war classloader, but your classes‎ country.yyy.xxx haven't access to org/joda/time/tz/DefaultNameProvider – Ensample

The classes (in this example InitialFixingDateTimeUtil) have access to org.joda.time.DateTimeZone and org.joda.time.tz.DefaultNameProvider. Mostly the code works fine, so in my opionion it is not an import problem. But sometimes (we do not know why) it ends in an java.lang.NoClassDefFoundError, even if it worked 1 minute before. – Juvenility

when the error occures, we can look in the file C:/Workarea/Projects/xxx-trunk/xxx-ear/target/xxx.ear/xxxEJB.jar/ and we can see, the class-file is definitely there. but it seams like JBoss cannot access it. – Juvenility

In Production the error never happens. a big difference is, that we deploy the application via copy the xxx.ear in the jboss-as-server deploy folder. – Juvenility

But file:/C:/Workarea/Projects/xxx-‎trunk/xxx/target/classes/ definitely is not a part of ear file. Who loads it? with what classloader? If this issue is really showstopper, you can switch to manual deploy via ant/mvn package and copy to jboss deploy folder. – Ensample

Just a thought - if you are using EAP, you should consider opening support tickets with JBoss. They probably have better expertise at this. – Workmanship

@user1516873: We will try to deploy via ant/maven and copy the artifact xxx.ear directly into jboss-as-server deploy-folder. – Juvenility

@CoolBeans: we are using EAP, and i forwarded the behavour already to the guys who have knowlegde about redhat support processes. thanks. – Juvenility

When we deploy the xxx.ear via copying the file into the JBoss-AS-Deploy-Folder it seams to work fine. This issue could be a IntelliJ IDEA problem, when deploying via the JBoss-Plugin. Or probably the mechanism the plugin uses to do that. – Juvenility

Answer 1

The problem just happens when we deploy with IntelliJ IDEA JBoss-Run-Configuration.

Thanks to user1516873 for the hint, to deploy manually via ANT/Maven.

We have tried it out.

We introduced an ANT-Task to just copy the artefact xxx.ear into the JBoss-Deploy-Folder. Since then it works fine, and we haven't got an java.lang.NoClassDefFoundError yet.

build.xml looks like this:

<project name="xxx-deploy">
    <!-- Copy Artifact to JBoss-Deploy-Folder -->
    <target name="xxx-ear-deploy">
        <copy file="C:/Workarea/Projects/xxx-trunk/xxx-ear/target/xxx.ear"
              todir="C:/Workarea/_tools/jboss-eap-5.1.0-xxx/jboss-as/server/default/deploy"/>
    </target>
</project>

Juvenility

Answer 2

It may be due to jar conflict inside the binary(xx.ear). You can use the below code to find out jar duplication in the lib folder of ear folder

public class JarSearch {

public static final String NL = System.getProperty("line.separator");

public static String directory = new String();
public static File file;
public static Writer output;

public static void main(String args[]) {
    /*if (args.length != 2){
        System.out.println("You must provide 1. search directory and 2. search string");
        System.exit(0);
    }*/
    JarSearch search = new JarSearch();
    try{
        String dir = "E://lib";//path of lib folder
        directory = dir;
        file = new File(directory);
        String parentPath = file.getParent();
        parentPath += "\\JarSearchResult.txt";
        file = new File(parentPath);
        //output = new BufferedWriter(new FileWriter(file));
        String searchStr = "DES.class";// name of class file to b searched
        search.searchDir(dir, searchStr);//"C:\\Program Files\\IBM\\Application Developer\\plugins\\com.ibm.etools.ctc.binding.java\\runtime\\ctcjava.jar");
    }catch(IOException ioe){
        System.out.println(ioe.toString());
    }
}

public static void searchDir(String dirName, String searchStr)throws IOException{
    File dir = new File(dirName);
    if (!dir.isDirectory()){
        throw new IOException(dir+" is not a directory !!!");
    }
    File files[] = dir.listFiles();
    for (int i=0; i<files.length; i++){
        if (files[i].isDirectory()){
            searchDir(files[i].getAbsolutePath(), searchStr);
        }else{
            if (files[i].getName().endsWith(".jar")){
                searchFile(files[i].getAbsolutePath(), searchStr);
            }
        }
    }
    //output.close();
}

public static void searchFile(String fileName, String searchStr)throws IOException{
    //System.out.println("File Absolute Path :: " + fileName);
    boolean outputMessage = false;
    StringBuffer buf = new StringBuffer();
    int index1 = fileName.lastIndexOf("\\");
    int index2 = fileName.lastIndexOf("/");
    //System.out.println(index1 + " :: " + index2 + " ===>>> " + NL);
    int index = -1;
    if (index1 > index2){
        index = index1;
    } else {
        index = index2;
    }
    buf.append(NL);

    buf.append("DIR: ");
    buf.append(fileName.substring(0, index));
    buf.append(NL);

    buf.append("JAR: ");
    buf.append(fileName.substring(index+1));
    buf.append(NL);

    searchStr = searchStr.toLowerCase();

    JarFile jar = new JarFile(fileName);
    Enumeration e = jar.entries();
    while (e.hasMoreElements()){
        JarEntry entry = (JarEntry)e.nextElement();
        String strEntry = entry.getName().toLowerCase();
        //System.out.println(strEntry);
        if (strEntry.indexOf(searchStr) > -1){
            outputMessage = true;
            buf.append(entry.getName());
            buf.append(NL);
        }
    }
    if (outputMessage){
        System.out.println(buf);
        //output.write(buf.toString());
    }
}

}

Rabbit

I tried it out. But i guess it is another thematic. The thing is, after a while the application is running a .class file is missing. – Juvenility