JPMS: --add-opens doesn't work for java.lang.reflect.InaccessibleObjectException

Question

I use Java 14 with Jetty 9.4 and Weld-servlet-shaded-3.1 and I get this exception:

Caused by: java.lang.reflect.InaccessibleObjectException: 
Unable to make protected final java.lang.Class   
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
throws java.lang.ClassFormatError accessible: module java.base does 
not "opens java.lang" to module weld.servlet.shaded.

According to answer here: https://mcmap.net/q/118297/-how-to-solve-inaccessibleobjectexception-quot-unable-to-make-member-accessible-module-a-does-not-39-opens-package-39-to-b-quot-on-java-9 I tried to add

--add-opens java.base/java.lang=ALL-UNNAMED 

and

--add-opens java.base/java.lang=weld.servlet.shaded 

But no one helped. Could anyone say how to fix it?

EDIT 1
I found out that this problems appears in Java 12,13,14 OpenJDK/Oracle. However, if I use Java 11 OpenJDK/Oralce everything works fine and I even don't add any --add-opens. How to explain it?

EDIT 2
I've checked Jetty pom and Weld pom they both have the same properties:

<compiler.source>1.8</compiler.source>
<compiler.target>1.8</compiler.target>

EDIT 3
Java version

$ java -version
openjdk version "14" 2020-03-17
OpenJDK Runtime Environment (build 14+36-1461)
OpenJDK 64-Bit Server VM (build 14+36-1461, mixed mode, sharing)

Start application

$java .... --add-opens java.base/java.lang=weld.servlet.shaded ...

And this is stack trace

09:58:41.243 [RMI TCP Connection(2)-127.0.0.1] INFO  org.jboss.weld.environment.servletWeldServlet - WELD-ENV-001008: Initialize Weld using ServletContainerInitializer
09:58:41.293 [RMI TCP Connection(2)-127.0.0.1] INFO  org.jboss.weld.Version - WELD-000900: 3.1.0 (Final)
09:58:41.468 [RMI TCP Connection(2)-127.0.0.1] WARN  org.jboss.weld.environment.servletWeldServlet - WELD-ENV-001004: Found both WEB-INF/beans.xml and WEB-INF/classes/META-INF/beans.xml. It is not portable to use both locations at the same time. Weld is going to use: file:/home/<SOME TEXT DELETED>/webapp/WEB-INF/beans.xml
09:58:41.479 [RMI TCP Connection(2)-127.0.0.1] INFO  org.jboss.weld.Bootstrap - WELD-ENV-000014: Falling back to Java Reflection for bean-discovery-mode="annotated" discovery. Add org.jboss:jandex to the classpath to speed-up startup.
09:58:41.550 [RMI TCP Connection(2)-127.0.0.1] WARN  org.eclipse.jetty.annotations.ServletContainerInitializersStarter - 
java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to module weld.servlet.shaded
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:349) ~[?:?]
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:289) ~[?:?]
    at java.lang.reflect.Method.checkCanSetAccessible(Method.java:196) ~[?:?]
    at java.lang.reflect.Method.setAccessible(Method.java:190) ~[?:?]
    at org.jboss.weld.util.bytecode.ClassFileUtils$1.run(ClassFileUtils.java:88) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
    at org.jboss.weld.util.bytecode.ClassFileUtils.makeClassLoaderMethodsAccessible(ClassFileUtils.java:64) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.bootstrap.WeldStartup.startContainer(WeldStartup.java:220) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.bootstrap.WeldBootstrap.startContainer(WeldBootstrap.java:72) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.bootstrap.WeldBootstrap.startContainer(WeldBootstrap.java:67) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.environment.servlet.WeldServletLifecycle.initialize(WeldServletLifecycle.java:182) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.environment.servlet.EnhancedListener.onStartup(EnhancedListener.java:62) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:140) ~[jetty-plus-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:64) [jetty-annotations-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346) [jetty-servlet-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) [jetty-webapp-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) [jetty-webapp-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) [jetty-server-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) [jetty-servlet-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) [jetty-webapp-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.14.v20181114.jar:?]
    <SOME TEXT DELETED>
09:58:41.555 [RMI TCP Connection(2)-127.0.0.1] WARN  org.eclipse.jetty.webapp.WebAppContext - Failed startup of context o.e.j.w.WebAppContext@6a329710{/,file:///home//<SOME TEXT DELETED>/webapp/,UNAVAILABLE}{/home/<SOME TEXT DELETED>.war}
java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to module weld.servlet.shaded
    at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:69) ~[jetty-annotations-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:346) ~[jetty-servlet-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) ~[jetty-webapp-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) ~[jetty-webapp-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) [jetty-webapp-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.4.14.v20181114.jar:?]
    <SOME TEXT DELETED>
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to module weld.servlet.shaded
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:349) ~[?:?]
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:289) ~[?:?]
    at java.lang.reflect.Method.checkCanSetAccessible(Method.java:196) ~[?:?]
    at java.lang.reflect.Method.setAccessible(Method.java:190) ~[?:?]
    at org.jboss.weld.util.bytecode.ClassFileUtils$1.run(ClassFileUtils.java:88) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]
    at org.jboss.weld.util.bytecode.ClassFileUtils.makeClassLoaderMethodsAccessible(ClassFileUtils.java:64) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.bootstrap.WeldStartup.startContainer(WeldStartup.java:220) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.bootstrap.WeldBootstrap.startContainer(WeldBootstrap.java:72) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.bootstrap.WeldBootstrap.startContainer(WeldBootstrap.java:67) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.environment.servlet.WeldServletLifecycle.initialize(WeldServletLifecycle.java:182) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.jboss.weld.environment.servlet.EnhancedListener.onStartup(EnhancedListener.java:62) ~[weld-servlet-shaded-3.1.0.Final.jar:?]
    at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:140) ~[jetty-plus-9.4.14.v20181114.jar:?]
    at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:64) ~[jetty-annotations-9.4.14.v20181114.jar:?]
    ... 31 more

EDIT 4
I found out that different behavior happens because of weld. This is from piece from ClassFileUtils:

public static void makeClassLoaderMethodsAccessible() {
    // the AtomicBoolean make sure this gets invoked only once as WeldStartup is triggered per deployment
    if (classLoaderMethodsMadeAccessible.compareAndSet(false, true)) {
        try {
            AccessController.doPrivileged(new PrivilegedExceptionAction<Object>() {
                public Object run() throws Exception {
                    Class<?> cl = Class.forName("java.lang.ClassLoader");
                    final String name = "defineClass";

                    defineClass1 = cl.getDeclaredMethod(name, String.class, byte[].class, int.class, int.class);
                    defineClass2 = cl.getDeclaredMethod(name, String.class, byte[].class, int.class, int.class, ProtectionDomain.class);

                    // First try with Unsafe to avoid illegal access
                    try {
                        // get Unsafe singleton instance
                        Field singleoneInstanceField = Unsafe.class.getDeclaredField("theUnsafe");
                        singleoneInstanceField.setAccessible(true);
                        Unsafe theUnsafe = (Unsafe) singleoneInstanceField.get(null);

                        // get the offset of the override field in AccessibleObject
                        long overrideOffset = theUnsafe.objectFieldOffset(AccessibleObject.class.getDeclaredField("override"));

                        // make both accessible
                        theUnsafe.putBoolean(defineClass1, overrideOffset, true);
                        theUnsafe.putBoolean(defineClass2, overrideOffset, true);
                        return null;
                    } catch (NoSuchFieldException e) {
                        // This is JDK 12+, the "override" field isn't there anymore, fallback to setAccessible()
LINE 88                 defineClass1.setAccessible(true);
                        defineClass2.setAccessible(true);
                        return null;
                    }
                }
            });
        } catch (PrivilegedActionException pae) {
            throw new RuntimeException("cannot initialize ClassPool", pae.getException());
        }
    }
}

as you see at line 88 ti calls setAccessible(true) method that is blocked by JPMS. So, the last question is it possible to solve this problem by JPMS settings, taking into consideration that that Weld and Jetty are in child JPMS layer?

Vivavivace

Did you ensure Jetty and Weld versions in use are compatible with Java version above 11? Considering no silly mistakes, that seems like to be the only possibility for failure here(even after adding the --add-opens VM arg). – Hawken

@Hawken Please, see edit 2. I hope I have no silly mistakes - I have my java application and launch it without any code modification with different JDK. With Java 11 it works, with Java 12,13,14 it doesn't. – Vivavivace

@Hawken Please, see edit 3. – Vivavivace

It looks like there is code in module weld.servlet.shaded trying to hack into ClassLoader.defineClass. The supported API for injecting classes into the same runtime package as another class is Lookup.defineClass. You should be able to workaround it with --add-opens java.base/java.lang=weld.servlet.shaded but only if weld.servlet.shaded is on the module path. You seem to be using that option so it makes me wondering if weld.servlet.shaded is in a child layer (hard to tell from the decorated stack trace) – Doris

@AlanBateman Thank you very much for your answer. Yes, I have several layers and Jetty and Weld are together in one child layer. What should I do then as --add-opens java.base/java.lang=weld.servlet.shaded doesn't work for me. And another important question - could you say why it works for java 11 and doesn't work for java 12-14? – Vivavivace

The original post mentions it works with JDK 11, are you sure about that? I can't explain that observation. In any case, I think someone should submit a bug to Weld on this as hacking into ClassLoader.defineClass will break completely once the the core modules are fully encapsulated (meaning it won't work on the class path either) – Doris

@AlanBateman Yes, it works with JDK 11!!! I've checked 100 times and now I work with JDK 11 version. I was surprised too as I thought that JPMS doesn't have to change from version to version. Maybe it is bug in JDK? – Vivavivace

Are you 100% sure that weld.servlet.shaded is not on the class path in the JDK 11 test? Also wondering if there is any possibility that there is code in Weld that detects when it is on JDK 11 and it does something different. – Doris

@AlanBateman Yes, I am 100 sure that weld.servlet.shaded is not on the class path. I don't use class path (cp) at all. I start my application this way: ` --add-modules ALL-DEFAULT --module-path "$MODULE_PATH" --add-opens java.base/java.lang=weld.servlet.shaded -m foo.core/foo.core.launcher.Launcher ` where MODULE_PATH contains path to initial modules. When application has started it creates another layer and adds to this layer jetty, weld etc. This application is freeware (not open source) if you want I can share it with you. With java 11 it works, but java > 11 it doesnt – Vivavivace

Jetty 10.x is where the Eclipse Jetty JPMS focus is. – Peppercorn

@AlanBateman Please, see edit 4 and if you answer it the question can be closed. – Vivavivace

The AccessibleObject.override field is a private field that is highly security sensitive. It is hidden from core reflection in recent JDK releases so I think this explains why you see a difference. That said, the maintainers of ClassUtils should not be hacking into this private field and setting it with Unsafe. It is unsupportable to be hacking into private fields like this. If the use-case is injecting a class into the same run-time package as another class then they should use Lookup.defineClass. – Doris

@Vivavivace if you have time, it would be good to submit a bug to the Weld project on this. – Doris

@AlanBateman Here it is: issues.redhat.com/browse/WELD-2620 – Vivavivace

Answer 1

As I found out there were two problems:

1. Weld ClassFileUtils.makeClassLoaderMethodsAccessible() works differently with JDK11 and JDK12+.
2. --add-opens as CLI parameter doesn't work for dynamically created layers JPMS? See details here

Vivavivace