EAP 7.3: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more - only in standalone mode

Question

we've got a web app (war) that deploys successfully to JBoss EAP 7.3 in domain mode (using the default profile), however deployment in standalone mode (config based on the default standalone.xml) it fails with:

14:47:40,342 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 88) MSC000001: Failed to start service jboss.deployment.unit."app.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."app.war".undertow-deployment: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
        at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:254)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
        ... 8 more
Caused by: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
        at javax.el.FactoryFinder.newInstance(FactoryFinder.java:64)
        at javax.el.FactoryFinder.find(FactoryFinder.java:103)
        at javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:140)
        at javax.el.ELUtil.getExpressionFactory(ELUtil.java:185)
        at javax.el.ELManager.getExpressionFactory(ELManager.java:38)
        at org.apache.jasper.runtime.JspApplicationContextImpl.<init>(JspApplicationContextImpl.java:49)
        at org.apache.jasper.runtime.JspApplicationContextImpl.getInstance(JspApplicationContextImpl.java:78)
        at org.apache.jasper.runtime.JspFactoryImpl.getJspApplicationContext(JspFactoryImpl.java:218)
        at org.wildfly.extension.undertow.deployment.JspInitializationListener.contextInitialized(JspInitializationListener.java:51)
        at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
        at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:217)
        at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:186)
        at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
        at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1504)
        at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:252)
        ... 10 more
Caused by: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more from [Module "deployment.app.war" from Service Module Loader]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
        at javax.el.FactoryFinder.newInstance(FactoryFinder.java:48)
        ... 29 more

14:47:40,351 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "app.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"app.war\".undertow-deployment" => "java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
    Caused by: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
    Caused by: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
    Caused by: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more from [Module \"deployment.app.war\" from Service Module Loader]"}}

War and the application configuration are identical in both cases.

JBoss configuration is as close as possible in both cases. In domain mode it's based on the default profile with basically just a few resources added. In standalone mode it's based on the default standalone.xml with the same set of resources added. Therefore from this point of view I can't find any reason for the difference in behavior.

So far we've tracked this to tomcat-embed-el-9.0.39.jar which is included in our war due to some 3rd party dependency. This jar contains META-INF/services/javax.el.ExpressionFactory with the following content:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

org.apache.el.ExpressionFactoryImpl

Where the issue is obviously that the 1st line is read and processed as an actual entry (e.g. class name) even though it's just a comment. It's therefore similar to the following older issue that affected EAP 6: https://access.redhat.com/solutions/1270603 (Red Hat login required).

It's however strange that this occurs only in the standalone mode. Could this be considered a bug?

Note: we tested this at JBoss EAP 7.3.0 and 7.3.6 with the same results.

Revkah

Have you tried excluding that library? JBoss EAP ships with an EL implementation of it's own. – Akron

Yes then it of course deploys successfully. However my main concerns are: 1. There's a strange difference between the standalone and domain modes. 2. That the comment is incorrectly read as an actual content. – Revkah

When you deploy to domain, are you deploying to active servers? As to why the comment is read, that looks like an issue with the EL factory. – Akron

Yes we usually redeploy with the target servers stopped, however I just tried a clean deploy to the running server and it still completed successfully (in domain). – Revkah

That definitely seems odd. It should be getting processed through all the some processors. – Akron

I can't exactly give an answer, but I ran into a similar problem deploying a Spring Boot 2.5 application to Wildfly 23. The direct cause of the issue appears to be that the Tomcat EL implementation violates the spec from docs.oracle.com/javaee/7/api/javax/el/… which says: Use the Services API (as detailed in the JAR specification). If a resource with the name of META-INF/services/javax.el.ExpressionFactory exists, then its first line, if present, is used as the UTF-8 encoded name of the implementation class. and its first line is the Apache text – Cotidal

Answer 1

This is bug/mismatch between the ExpressionFactory implementation and the ExpressionFactory provider file (META-INF/services/javax.el.ExpressionFactory).

The ExpressionFactory.newInstance() method is expected to use the JAR services API to find the name of an implementation class:

Use the Services API (as detailed in the JAR specification). If a resource with the name of META-INF/services/javax.el.ExpressionFactory exists, then its first line, if present, is used as the UTF-8 encoded name of the implementation class.

This description is somewhat ambiguous/inconsitent:

• it claims that the first line of the file should be used as the class name;
• on the other hand it says it should use the Services API which is expected to ignore comments.

While decompiling my ExpressionFactory implementation, I find:

private static String getClassNameServices(final ClassLoader tccl) {
  InputStream is = null;
  if (tccl == null) {
    is = ClassLoader.getSystemResourceAsStream("META-INF/services/javax.el.ExpressionFactory");
  }
  else {
    is = tccl.getResourceAsStream("META-INF/services/javax.el.ExpressionFactory");
  }
  if (is != null) {
    String line = null;
    BufferedReader br = null;
    InputStreamReader isr = null;
    try {
      isr = new InputStreamReader(is, "UTF-8");
      br = new BufferedReader(isr);
      line = br.readLine();
      if (line != null && line.trim().length() > 0) {
        return line.trim();
      }
    }
    // [...]
  }
}

This implementation does not ignore comments. My META-INF/services/javax.el.ExpressionFactory (in jasper-el.jar) actually looks like:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

org.apache.el.ExpressionFactoryImpl 

As a consequence, it tries to load a class named "# Licensed to the Apache Software Foundation (ASF) under one or more" which fails.

The relevant bug is bug 64097 and a fix is found in commit 9ed78bcc80565292cd9a343f4e6d0c2db305494e.

A (horrible) quick fix would be to patch the META-INF/services/javax.el.ExpressionFactory file like this:

mkdir -p META-INF/services/
echo org.apache.el.ExpressionFactoryImpl > META-INF/services/javax.el.ExpressionFactory
7z d ./jasper-el.jar META-INF/services/javax.el.ExpressionFactory
jar uf ./jasper-el.jar META-INF/services/javax.el.ExpressionFactory

Arellano

Thanks for pointing to the right direction. Just encountered this unpleasant issue with my local Tomcat version 9.0.21. The easiest solution was to just upgrade the Tomcat to the latest (as of now) version 9.0.63 and the issue was resolved. – Klaraklarika

Answer 2

Upgrading to Tomcat 9.0.63 resolved the issue.

User informatik01's comment on another answer set me on that path -- thanks to them!

Divorcee