solution for java.lang.VerifyError in tomcat 7.0.23/jdk 1.6.x?

Question

When I deployed a maven 3.0.3 built webapp into tomcat 7.0.23 on Linux (java 1.6.x) and accessed posted my credentials in the login page, I got the following error. pom.xml references servlet 2.5, jsp 2.1 and JSTL 1.2.

Why would I get this Error ? what can I do to avoid it ?

I have other .war files deployed on the same tomcat instance and never faced this issue in those apps.

The 1st JSP(login.jsp) that I did a http GET didn't throw an error. The POST from the 1st JSP to the 2nd Jsp(ChLogin.jsp) encountered the problem.

===========================

root cause

java.lang.VerifyError: (class: org/apache/jsp/ChLogin_jsp, method: _jspService signature: (Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V) Inconsistent stack height 0 != 1
    java.lang.Class.getDeclaredConstructors0(Native Method)
    java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
    java.lang.Class.getConstructor0(Class.java:2699)
    java.lang.Class.newInstance0(Class.java:326)
    java.lang.Class.newInstance(Class.java:308)
    org.apache.jasper.servlet.JspServletWrapper.getServlet(JspServletWrapper.java:172)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
    org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
    org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

UPDATE 1
the WEB-INF\lib of the webapp does NOT contain servlet, jsp or JSTL jars. It contains many application dependencies and spring jars (we use spring MVC).

UPDATE 2
We don't pre-compile JSPs. That is a goal for another day, though.

UPDATE 3
Discovered that this problem was not in tomcat 7.0.12, 7.0.14, 7.0.16 but started with 7.0.19. Posted to tomcat users mailing list. got a reply that it was perhaps the eclipse JDT compiler at fault. Verified that is indeed the case by replacing the compiler in tomcat 7.0.23/27 with tomcat 7.0.16's JDT compiler and LO & BEHOLD, all is well. I am planning to write to the Eclipse JDT compiler team(if there is one) and post about this error.

Lorielorien

Do you use any bytecode rewriting, aspect frameworks or java agents that might process the compiled jsps? – Sedgewick

We are using the spring MVC library -- but these JSPs have not been configured for spring. Spring is setup to use a different URL path. – Lorielorien

I have the same .WAR file to Tomcat 5.5 and it works just fine. So, what am I missing ? – Lorielorien

I'm getting the same VerifyError after changing the pom.xml for JSTL library to use scope provided. What else should I do ? Thank you. – Lorielorien

Just to be sure, the WAR file does not contain servlet/JSP/JSTL jar files. Also, I don't precompile any jsp files. The appserver needs to compile the JSP files. – Lorielorien

Please post a small WAR file with the same trouble. It will help to investigate/reproduce troubles. – Neu

Your problem is complex. We should have a way to reproduce it easily if your want a relevant anwer. Please publish a small WAR or a SSCCE to build from source. – Neu

I wish I could post a small WAR with the same trouble. I will try though. – Lorielorien

At least please publish "verbose:class" output, that stack trace only is not enough. By the way, you will have to do the effort to reproduce the issue with a small set of code (even the smallest set as possible), even a medium may not answer from so limited information. – Neu

Any chance you could post the full source of the JSP? – Howlet

Or the generated Servlet Java source file from Tomcat work directory ? – Neu

unfortunately, proprietary company code -- I won't be able to post it. – Lorielorien

how could I have posted "verbose:class" output ? – Lorielorien

Answer 1

The error is quite obvious: there is a bug in the jasper creating the bytecode. The generated code doesn't clear the stack frame propery Inconsistent stack height 0 != 1

You may try to shuffle the code, split into methods, move lines here and there and the bug might go away. Also can you post relevant info of the jsp servlet (matching *.jsp) from web.xml (usually somewhere in conf) to check compiler parameters?

Tamp

thanks. I'm using stock tomcat 7.0.27 (apache-*.tar.gz) and didn't change any of the web.xml. Other webapps deployed to the same tomcat don't have any issues. Why only "my" webapp ? – Lorielorien

Why only "my" webapp, read the answer again, probably it's a (rare) bug. I have no idea how web.xml looks on stock tomcat actually and usually tuning jasper is a good idea anyways. Try some of the options in the answer, i.e. extract methods add some local variable, etc and most likely the problem will go away. Your web-app is just unlucky. – Tamp

Jasper itself does not generate bytecode. It generates Java source files as standard Servlet that are compiled with Eclipse JDT compiler (which may be considered as reliable) – Neu

@YvesMartin, I know that, back in the day I even wrote custom handler to use either JBuilder internal compiler or Sun's javac V8. JSP servlet handles all *.jsp (by default) and does its stuff (it was possible to even trick jasper and use its local variables). I was curious if it uses JDT (and it's available in the classpath) or forks javac. Even JDT is reliable I have fallen on bugs, VerifyError is definitely a bug (bad bytecode) in a method body, otherwise (different class versions for instance) would be some other LinkageError like NoSuchMethodError, ClassFormatError or so. – Tamp

Already got VerifyError with JSP API in WAR and I am not alone: #8821588 – Neu

Another one: #7534378 – Neu

@YvesMartin, notice that VerifyError is not for the compiled servlet (org/apache/jsp/ ChLogin_jsp) but for class: org/apache/jasper/runtime/JspApplicationContextImpl and class: org/apache/jasper/compiler/JspUtil; the case is quite different. I'd ask the OP to just call [via web browser] the rest of the JSPs and see if any other VerifyError pops up. – Tamp

thanks for the input. I found that indeed the embedded eclipse JDT compiler that generates the code for the JSP was at fault. More details in the update 3 – Lorielorien

@anjanb, it's not the 1st bug in eclipse's JDT [we did have few issues submitted during the years - fav. warning for double==double - comparing equal blah, blah - hint doesn't work for NaN], if you can send over the JSP (generated java) I might even try my hand to fix the issue, as for immediate solution, you can switch to javac compiler in web.xml. Look for the jsp servelt doc for further info (you need it in your web-app) – Tamp

@Tamp : thanks. Either use javac or pre-compile the classes. The easiest option seems to be to use javac. – Lorielorien

Answer 2

This error means the JSP has been compiled for a specific Servlet/JSP/JSTL implementation but then Tomcat tries to load compiled classes with an other implementation/version available in ClassPath.

It probably happens because your WAR contains jar files with servlet, jsp and/or jstl API and/or implementation. As used versions match Tomcat 5.5 imlementation, it works there but fail with more recent implementations.

You have to set the scope to provided for such dependencies in your Maven POM to exclude that jars from WAR WEB-INF/lib.

To validate that on-the-fly compiled code does not load alternate dependencies, you should run Tomcat with -verbose:class. Latest lines before the error may help to guess what is wrong.

Update 2012/05/30 Do you have any jasper or javax.el or el-api jar in WEB-INF/lib ? If so, remove it/them as well.

It is also possible that a taglib you used is specifically compiled for Tomcat 5.5 Servlet 2.3 API and breaks when Tomcat 7.

Neu

thank you. "You have to set the scope to runtime for such dependencies in your Maven POM." Can you elaborate a lil more ? point me to some sample ? – Lorielorien

I link "scope" to Maven reference about dependency scope. I made a mistake - you have to use "provided" in your case. – Neu

For what it's worth, I really think @Yves has the right answer. Make sure that all your <dependency> entries for servlet, jsp, and jstl have <scope>provided</scope> inside of them. Try it and at least let us know if it worked. – Appreciate

What is the content of your WEB-INF/lib ? Have you pre-compiled your JSP with Jasper "jspc" ? – Neu

I have bunch of application related jars and a bunch of spring related jars. Does not contain servlet, jsp or jstl. No, I do not pre-compile JSPs. – Lorielorien

What about XML stuff like STAX, Xerces, Xalan... that are now included in JDK ? – Neu

The entire list of jars are generated/obtained by maven. Nope, none of the above stuff. – Lorielorien

this is proprietary code. cannot publish code. don't have permission. – Lorielorien

So please build a small SSCE from scratch with most of the libraries that are used. It fails with a single JSP, it should not be difficult to reproduce. – Neu

Modded this answer down: VerifyError has nothing to do with incorrect JSP versions and/or precompilation. VerifyError is a JVM error complaining that the .class file itself (or, rather, the bytecode) is bad. – Howlet

I am still waiting for any additional info (verbose:class) or even the bytecode itself why not - there are tools to read and validate its structure. The JSP generated Servlet source code should be available in temporary directory. Can we get it ? – Neu

Answer 3

Have you tried disabling the Split Verifier: -XX:-UseSplitVerifier

Here is a reference for JVM options: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

I have seen something similar with Java 7 and the VerifyError.

Omit

@Lorielorien No, that flag disables the verifier used by default in recent JDKs and falls back to an older version. I would only expect this to help if your tool chain is doing bytecode weaving somewhere along the line. – Blackington

at this point, the JSP to which we're POSTing http data is not a spring jsp and we're not really into any bytecode weaving any of our code; however, will try it out and let you know soon. – Lorielorien

Answer 4

Try printing the classpath (System.out.println("java.class.path");) Compare the classpath of the tomcat5 and tomcat7. Check what is the difference in the classpath.

Router

the old classpath had a lot more jars that are required. I'm trying to reduce the number of jars here. – Lorielorien

Yes this way you can have an idea of whats going wrong... ensure the classpath is same... – Router

Answer 5

This happened with me on Tomcat 6.0.35 and just in one jsp page, which has only:

<jsp:forward page="/sindicato/consultaServidor.jsp"/>

The solution was replace the jsp tag by jstl tag:

<c:redirect url="/sindicato/consultaServidor.jsp"/>

And don't ask why! Because we have more jsp pages with that jsp tag (jsp:forward) with no problem.

Soriano

Answer 6

java.lang.VerifyError:

Easy W/A

This is because of the built in tomcat compiler.

The Java compiler from Eclipse JDT in included as the default compiler. It is an advanced Java compiler which will load all dependencies from the Tomcat class loader, which will help tremendously when compiling on large installations with tens of JARs. On fast servers, this will allow sub-second recompilation cycles for even large JSP pages.

If any such issues is seen follow the below workaround

Apache Ant, which was used in previous Tomcat releases, can be used instead of the new compiler by simply removing the lib/ecj-*.jar file, and placing the ant.jar and ant-launcher.jar files from the latest Ant distribution in the lib folder.

Warthog