Lambda expressions and anonymous classes don't work when loaded as hidden classes
Asked Answered
L

2

11

I am trying to compile and load dynamically generated Java code during runtime. Since both ClassLoader::defineClass and Unsafe::defineAnonymousClass have serious drawbacks in this scenario, I tried using hidden classes via Lookup::defineHiddenClass instead. This works fine for all classes that I tried to load, except for those that call lambda expressions or contain anonymous classes.

Calling a lambda expression throws the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: tests/HiddenClassLambdaTest$LambdaRunner/0x0000000800c04400
    at tests.HiddenClassLambdaTest.main(HiddenClassLambdaTest.java:22)
Caused by: java.lang.ClassNotFoundException: tests.HiddenClassLambdaTest$LambdaRunner.0x0000000800c04400
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
    ... 1 more

Executing code that instantiates an anonymous class throws the following error:

Exception in thread "main" java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    tests/HiddenClassLambdaTest$LambdaRunner+0x0000000800c00400.run()V @5: invokespecial
  Reason:
    Type 'tests/HiddenClassLambdaTest$LambdaRunner+0x0000000800c00400' (current frame, stack[2]) is not assignable to 'tests/HiddenClassLambdaTest$LambdaRunner'
  Current Frame:
    bci: @5
    flags: { }
    locals: { 'tests/HiddenClassLambdaTest$LambdaRunner+0x0000000800c00400' }
    stack: { uninitialized 0, uninitialized 0, 'tests/HiddenClassLambdaTest$LambdaRunner+0x0000000800c00400' }
  Bytecode:
    0000000: bb00 1159 2ab7 0013 4cb1               

    at java.base/java.lang.ClassLoader.defineClass0(Native Method)
    at java.base/java.lang.System$2.defineClass(System.java:2193)
    at java.base/java.lang.invoke.MethodHandles$Lookup$ClassDefiner.defineClass(MethodHandles.java:2446)
    at java.base/java.lang.invoke.MethodHandles$Lookup$ClassDefiner.defineClassAsLookup(MethodHandles.java:2427)
    at java.base/java.lang.invoke.MethodHandles$Lookup.defineHiddenClass(MethodHandles.java:2133)
    at tests.HiddenClassLambdaTest.main(HiddenClassLambdaTest.java:25)

This is a short example that recreates the problem:

import java.lang.invoke.MethodHandles;

public class HiddenClassLambdaTest {
    /** This class is to be loaded and executed as hidden class */
    public static final class LambdaRunner implements Runnable {
        @Override public void run() {
            Runnable runnable = () -> System.out.println("Success");
            runnable.run();
        }
    }
    
    public static void main(String[] args) throws Throwable {
        // Path to the class file of the nested class defined above
        String nestedClassPath = HiddenClassLambdaTest.class.getTypeName().replace('.','/') + "$LambdaRunner.class";
        // Class file content of the LambdaRunner class
        byte[] classFileContents = HiddenClassLambdaTest.class.getClassLoader().getResourceAsStream(nestedClassPath).readAllBytes();
        Class<?> lambdaRunnerClass = MethodHandles.lookup().defineHiddenClass(classFileContents, true).lookupClass();
        Runnable lambdaRunnerInstance = (Runnable) lambdaRunnerClass.getConstructor().newInstance();
        lambdaRunnerInstance.run();
    }
}

I've already tried compiling and running the code with different JDKs, using different ways to create new instances of the hidden class, searching for bugs at https://bugs.openjdk.java.net/, messing with the bytecode itself and several other things. I am not an expert on Java internals, so I am not sure whether I have not understood the JEP that introduced hidden classes correctly.

Am I doing something wrong, is this just impossible or is this a bug?

Edit: The JEP states

Migration should take the following into account: To invoke private nestmate instance methods from code in a hidden class, use invokevirtual or invokeinterface instead of invokespecial. Generated bytecode that uses invokespecial to invoke a private nestmate instance method will fail verification. invokespecial should only be used to invoke private nestmate constructors.

This might be the problem for the anonymous class. Is there a way to compile the code such that invokespecial is avoided in the bytecode?

Lillylillywhite answered 22/2, 2022 at 16:26 Comment(6)
Well, the problem is that hidden classes can't be referenced by other classes. Lambdas are implemented by creating and loading a class at runtime that references the containing class. Which doesn't work when the containing class is hidden.Perreira
@JohannesKuhn it would work if that would have been a method reference: Runnable runnable = HiddenClassTest::print; static void print() { System.out.println("Success"); }Huckaback
If the method reference points to the hidden class, no. If it points somewhere else, it should work.Perreira
Unsafe::defineAnonymousClass is deprecated for removal and Lambdas don't work with that one either. Then I'm probably stuck with creating a class loader for each class that I load to make sure that they can be unloaded once they are no longer in use.Lillylillywhite
You could do some bytecode transformation before - use MethodHandles::invokeExact as new target.Perreira
If you have something concrete in mind that solves the problem (i.e. loads arbitrary code in a way so that it will be unloaded if no longer in use) I would be happy about a more detailed answer. All I got from experimenting with LambdaMetaFactory and MethodHandles was IncompatibleClassChangeErrors.Lillylillywhite
H
10

You can not turn arbitrary classes into hidden classes.

The documentation of defineHiddenClass contains the sentence

  • On any attempt to resolve the entry in the run-time constant pool indicated by this_class, the symbolic reference is considered to be resolved to C and resolution always succeeds immediately.

What it doesn’t spell out explicitly is that this is the only place where a type resolution ever ends up at the hidden class.

But it has been said unambiguously in bug report JDK-8222730:

For a hidden class, its specified hidden name should only be accessible through the hidden class's 'this_class' constant pool entry.

The class should not be accessible by specifying its original name in, for example, a method or field signature even within the hidden class.

Which we can check. Even a simple case like

public class HiddenClassLambdaTest {

    public static void main(String[] args) throws Throwable {
        byte[] classFileContents = HiddenClassLambdaTest.class
            .getResourceAsStream("HiddenClassLambdaTest$LambdaRunner.class")
            .readAllBytes();
        var hidden = MethodHandles.lookup()
            .defineHiddenClass(classFileContents, true, ClassOption.NESTMATE);
        Runnable lambdaRunnerInstance = (Runnable)hidden.findConstructor(
            hidden.lookupClass(), MethodType.methodType(void.class)).invoke();
        lambdaRunnerInstance.run();
    }

    static class LambdaRunner implements Runnable {
        LambdaRunner field = this;

        @Override
        public void run() {
        }
    }
}

will already fail. Note that it is a special case that the attempt to resolve the original class name LambdaRunner within the hidden class will not fail, as you used an existing class as template. So you get an IncompatibleClassChangeError or a VerifierError due to mismatches between the hidden class and the existing LambdaRunner class. When you don’t use a class definition of an existing class, you’d get a NoClassDefFoundError.

The same applies to

    static class LambdaRunner implements Runnable {
        static void method(LambdaRunner arg) {
        }

        @Override
        public void run() {
            method(this);
        }
    }

As the cited bug report said, neither field nor methods can refer to the hidden class in their signature.

A less intuitive example is

    static class LambdaRunner implements Runnable {
        @Override
        public void run() {
            System.out.println("" + this);
        }
    }

which will fail depending on the compiler and options, as when the StringConcatFactory is used, the behavior is like an invocation of a method having all non-constant parts as parameters and returning a String. So this is another case of having the hidden class in a method signature.


Lambda expressions are special, as a class like

    static class LambdaRunner implements Runnable {
        @Override
        public void run() {
            Runnable runnable = () -> System.out.println("Success");
            runnable.run();
        }
    }

gets compiled similar to

    static class LambdaRunner implements Runnable {
        @Override
        public void run() {
            Runnable runnable = LambdaRunner::lambdaBody;
            runnable.run();
        }
        private static void lambdaBody() {
            System.out.println("Success");
        }
    }

which doesn’t have the hidden class in the method signature, but has to refer to the method holding the body of the lambda expression as a MethodReference. Within the constant pool, the description of this method refers to its declaring class using the this_class entry. So it gets redirected to the hidden class as described in the documentation.

But the construction of the MethodType as part of the MethodReference does not use this information to load a Class like a class literal would do. Instead, it tries to load the hidden class through the defining class loader, which fails with the NoClassDefFoundError you have posted.

This seems to be related to JDK-8130087 which suggests that ordinary method resolution differs from the way, MethodType works, which can make MethodType fail where just invoking the method would work.

But it’s possible to demonstrate that even fixing this issue wouldn’t solve the general problem:

    static class LambdaRunner implements Runnable {
        @Override
        public void run() {
            var lookup = MethodHandles.lookup();
            var noArgVoid = MethodType.methodType(void.class);
            try {
                MethodHandle mh = LambdaMetafactory.metafactory(lookup, "run",
                    MethodType.methodType(Runnable.class), noArgVoid,
                    lookup.findStatic(LambdaRunner.class, "lambdaBody", noArgVoid),
                    noArgVoid).getTarget();
                System.out.println("got factory");
                Runnable runnable = (Runnable)mh.invokeExact();
                System.out.println("got runnable");
                runnable.run();
            }
            catch(RuntimeException|Error e) {
                throw e;
            }
            catch(Throwable e) {
                throw new AssertionError(e);
            }
        }
        private static void lambdaBody() {
            System.out.println("Success");
        }
    }

This bypasses the problem described above and calls the LambdaMetafactory manually. When being redefined as hidden class, it will print:

got factory
got runnable
Exception in thread "main" java.lang.NoClassDefFoundError: test/HiddenClassLambdaTest$LambdaRunner/0x0000000800c01400
    at test/test.HiddenClassLambdaTest.main(HiddenClassLambdaTest.java:15)
Caused by: java.lang.ClassNotFoundException: test.HiddenClassLambdaTest$LambdaRunner.0x0000000800c01400
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    ... 1 more

which shows that all obstacles have been circumvented, but when it comes to the actual invocation from the generated Runnable to the method holding the lambda body, it will fail due to the fact that the target class is hidden. A JVM with eager resolution of symbolic references might fail earlier, i.e. the example might not print got runnable then.

Unlike the old JVM anonymous classes, there is no way to link to a hidden class, not even from another hidden class.


The bottom line is, as said at the beginning, you can not turn arbitrary classes into hidden classes. Lambda expressions are not the only feature not working with hidden classes. It’s not a good idea to try and get surprised. Hidden classes should only be used in conjunction with bytecode generators carefully using only features known to work.

Horizontal answered 23/2, 2022 at 18:19 Comment(0)
P
1

As Holger pointed out, you can't load arbitrary classes with MethodHandles.Lookup.defineHiddenClass.

But you can perform some transformations on the class file to conform to the restrictions of a hidden class.

One such transformation is using MethodHandles::invokeExact as implementing method - so the lambda will capture a MethodHandle.

Using your code as base, I came up with this - gist with imports and w/o comments:

public class HiddenClassLambdaTest {
    /** This class is to be loaded and executed as hidden class */
    public static final class LambdaRunner implements Runnable {
        @Override public void run() {
            Runnable runnable = () -> System.out.println("Success");
            runnable.run();
        }
    }
    
    public static void main(String[] args) throws Throwable {
        // Path to the class file of the nested class defined above
        byte[] classFileContents = HiddenClassLambdaTest.class
                .getResourceAsStream("HiddenClassLambdaTest$LambdaRunner.class").readAllBytes();
        
        classFileContents = processLambdas(classFileContents);
        MethodHandles.Lookup hiddenClass = MethodHandles.lookup().defineHiddenClass(classFileContents, true);
        Runnable lambdaRunnerInstance = (Runnable) (
                hiddenClass.findConstructor(hiddenClass.lookupClass(), methodType(void.class))
                ).asType(methodType(Runnable.class)).invokeExact();
        lambdaRunnerInstance.run();
    }

We start of like your example code - except we process the classFileContents using ASM before passing it to MethodHandles.Lookup.defineHiddenClass.

    public static CallSite metafactory(MethodHandles.Lookup l, String name, MethodType mt,
            MethodType interfaceType, MethodHandle mh, MethodType dynamicMethodType) throws Throwable {
        MethodHandle invoker = MethodHandles.exactInvoker(mh.type());
        if (mt.parameterCount() == 0) {
            // Non-capturing lambda
            mt = mt.appendParameterTypes(MethodHandle.class);
            CallSite cs = LambdaMetafactory.metafactory(l, name, mt, interfaceType, invoker, dynamicMethodType);
            Object instance = cs.dynamicInvoker().asType(methodType(Object.class, MethodHandle.class)).invokeExact(mh);
            return new ConstantCallSite(MethodHandles.constant(mt.returnType(), instance));
        } else {
            // capturing
            MethodType lambdaMt = mt.insertParameterTypes(0, MethodHandle.class);
            CallSite cs = LambdaMetafactory.metafactory(l, name, lambdaMt, interfaceType, invoker, dynamicMethodType);
            return new ConstantCallSite(cs.dynamicInvoker().bindTo(mh));
        }
    }

As part of the trick we replace all invokedynamic instructions that point to LambdaMetafactory.metafactory with HiddenClassLambdaTest.metafactory.
There we replace the implementation method with MethodHandles.invokeExact, then add the original MethodHandle as part of the captured arguments.

You may want to move that method to a different class.

    public static CallSite altMetafactory(MethodHandles.Lookup l, String name, MethodType mt, Object... args) {
        throw new UnsupportedOperationException("Not Implemented");
    }

Same deal with LambdaMetafactory.altMetafactory. Except I did not implement it. ¯\(ツ)

    private static byte[] processLambdas(byte[] bytes) {
        ClassReader cr = new ClassReader(bytes);
        ClassWriter cw = new ClassWriter(cr, 0);
        
        ClassVisitor cv = cw;
        cv = new LambdaTransformer(cv);
        
        cr.accept(cv, 0);
        return cw.toByteArray();
    }

Just the usual boilerplate for ASM transformations.
Because I anticipate that I have to stack the transformations, I wrote it that way.

    private static class LambdaTransformer extends ClassVisitor {
        LambdaTransformer(ClassVisitor parent) {
            super(ASM9, parent);
        }
        
        @Override
        public MethodVisitor visitMethod(int access, String name, String descriptor,
                String signature, String[] exceptions) {
            return new LambdaMethodTransformer(super.visitMethod(access, name, descriptor, signature, exceptions));
        }
        
        private static class LambdaMethodTransformer extends MethodVisitor {
            public LambdaMethodTransformer(MethodVisitor parent) {
                super(ASM9, parent);
            }
            
            private static final ClassDesc CD_LambdaMetafactory = LambdaMetafactory.class.describeConstable().orElseThrow();
            private static final ClassDesc CD_HiddenLambdaTest = HiddenClassLambdaTest.class.describeConstable().orElseThrow();
            
            private static final DirectMethodHandleDesc LMF_FACTORY = ofCallsiteBootstrap(
                    CD_LambdaMetafactory, "metafactory", CD_CallSite, CD_MethodType, CD_MethodHandle, CD_MethodType);
            private static final DirectMethodHandleDesc LMF_ALTFACTORY = ofCallsiteBootstrap(
                    CD_LambdaMetafactory, "altMetafactory", CD_Object.arrayType());
            
            private static final Handle MY_FACTORY = toASM(ofCallsiteBootstrap(CD_HiddenLambdaTest, "metafactory", CD_CallSite, CD_MethodType, CD_MethodHandle, CD_MethodType));
            private static final Handle MY_ALTFACTORY = toASM(ofCallsiteBootstrap(CD_HiddenLambdaTest, "altMetafactory", CD_CallSite, CD_Object.arrayType()));
            
            @Override
            public void visitInvokeDynamicInsn(String name, String descriptor,
                    Handle bootstrapMethodHandle, Object... bootstrapMethodArguments) {
                MethodHandleDesc h = fromASM(bootstrapMethodHandle);
                if (h.equals(LMF_FACTORY)) {
                    super.visitInvokeDynamicInsn(name, descriptor,
                            MY_FACTORY, bootstrapMethodArguments);
                } else if (h.equals(LMF_ALTFACTORY)) {
                    super.visitInvokeDynamicInsn(name, descriptor, MY_ALTFACTORY, bootstrapMethodArguments);
                } else {
                    super.visitInvokeDynamicInsn(name, descriptor, bootstrapMethodHandle, bootstrapMethodArguments);
                }
            }

That was the important part - check if the bootstrap method of an invokedynamic instruction is LambdaMetafactory.metafactory or LambdaMetafactory.altMetafactory and replace it with HiddenClassLambdaTest.metafactory or HiddenClassLambdaTest.altMetafactory respectively.

            private static MethodHandleDesc fromASM(Handle h) {
                return MethodHandleDesc.of(Kind.valueOf(h.getTag(), h.isInterface()),
                        ClassDesc.ofDescriptor("L" + h.getOwner() + ";"),
                        h.getName(), h.getDesc());
            }
            
            private static Handle toASM(DirectMethodHandleDesc desc) {
                return new Handle(desc.refKind(), toInternal(desc.owner()), desc.methodName(), desc.lookupDescriptor(), desc.isOwnerInterface());
            }
            
            private static String toInternal(TypeDescriptor.OfField<?> desc) {
                String d = desc.descriptorString();
                if (d.charAt(0) != 'L') {
                    throw new IllegalArgumentException("Not a valid internal type: " + d);
                }
                return d.substring(1, d.length() - 1); // Strip "L" + ";"
            }
        }
    }
}

At the end some helper methods which allow me to easily convert between the ASM types and the java.lang.constant.* types.
I prefer to use the java.lang.constant.* API - those helper methods are usually statically imported from a utility class.


But this is only half of the story.
See Holger's answer for many more things that can go wrong.
Some of them have an "easy" solution, others may be more complicated.

For example, to get rid of fields that reference the class, replace the fields type with a superclass (java.lang.Object for example) and inject a checkcast instruction after each getfield or getstatic of those fields.

To get them out of the method signatures - you also have to change the signatures at the call sites.

Perreira answered 24/2, 2022 at 8:51 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.