Compiling a Java Class in memory with `lombok` annotations and Java JDK 8

Question

I'm trying to retrieve the description of a few Java Beans from an XML file. I'd like to annotate them with @Data from project lombok to automatically include constructor, equals, hashCode, getters, setters and toString. I'd like to compile them in memory, generate a few instances (with data from the same XML file) and add them to Drools to eventually do some reasoning on that data.

Unfortunately, I cannot compile those classes and so I am asking for your help!

The following code shows how to programmatically compile Java classes in memory:

package example;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import javax.tools.JavaCompiler;
import javax.tools.JavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.ToolProvider;

public class Simple {

    public static void main(String[] args) throws Exception {
        String name = "Person";
        String content = //
            "public class " + name + " {\n" + //
            "    @Override\n" + //
            "    public String toString() {\n" + //
            "        return \"Hello, world!\";\n" + //
            "    }\n" + //
            "}\n";
        System.out.println(content);

        JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
        JavaFileManager manager = new MemoryFileManager(compiler.getStandardFileManager(null, null, null));

        List<String> options = new ArrayList<String>();
        options.addAll(Arrays.asList("-classpath", System.getProperty("java.class.path")));

        List<JavaFileObject> files = new ArrayList<JavaFileObject>();
        files.add(new MemoryJavaFileObject(name, content));

        compiler.getTask(null, manager, null, options, null, files).call();

        Object instance = manager.getClassLoader(null).loadClass(name).newInstance();
        System.out.println(instance);
    }

}

where MemoryFileManager is:

package example;

import java.io.IOException;
import java.security.SecureClassLoader;

import javax.tools.FileObject;
import javax.tools.ForwardingJavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.JavaFileObject.Kind;
import javax.tools.StandardJavaFileManager;

public class MemoryFileManager extends ForwardingJavaFileManager<StandardJavaFileManager> {

    private MemoryJavaClassObject object;

    public MemoryFileManager(StandardJavaFileManager manager) {
        super(manager);
    }

    @Override
    public ClassLoader getClassLoader(Location location) {
        return new SecureClassLoader() {
            @Override
            protected Class<?> findClass(String name) throws ClassNotFoundException {
                byte[] b = object.getBytes();
                return super.defineClass(name, object.getBytes(), 0, b.length);
            }
        };
    }

    @Override
    public JavaFileObject getJavaFileForOutput(Location location, String name, Kind kind, FileObject sibling) throws IOException {
        object = new MemoryJavaClassObject(name, kind);
        return object;
    }

}

and MemoryJavaClassObject is:

package example;

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;

import javax.tools.SimpleJavaFileObject;

public class MemoryJavaClassObject extends SimpleJavaFileObject {

    protected final ByteArrayOutputStream stream = new ByteArrayOutputStream();

    public MemoryJavaClassObject(String name, Kind kind) {
        super(URI.create("string:///" + name.replace('.', '/') + kind.extension), kind);
    }

    public byte[] getBytes() {
        return stream.toByteArray();
    }

    @Override
    public OutputStream openOutputStream() throws IOException {
        return stream;
    }

}

and finally MemoryJavaFileObject is:

package example;

import java.net.URI;

import javax.tools.SimpleJavaFileObject;

public class MemoryJavaFileObject extends SimpleJavaFileObject {

    private CharSequence content;

    protected MemoryJavaFileObject(String className, CharSequence content) {
        super(URI.create("string:///" + className.replace('.', '/') + Kind.SOURCE.extension), Kind.SOURCE);
        this.content = content;
    }

    @Override
    public CharSequence getCharContent(boolean ignoreEncodingErrors) {
        return content;
    }

}

If I run the example in the first code block, I get the following output, as expected:

public class Person {
    @Override
    public String toString() {
        return "Hello, world!";
    }
}

Hello, world!

Now, if I add the lombok.jar into my project and I include the following example:

package example;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import javax.tools.JavaCompiler;
import javax.tools.JavaFileManager;
import javax.tools.JavaFileObject;
import javax.tools.ToolProvider;

public class Lombok {

    public static void main(String[] args) throws Exception {
        String name = "Person";
        String content = //
            "import lombok.Data;\n" + //
            "public @Data class " + name + " {\n" + //
            "    private String name;\n" + //
            "}\n";
        System.out.println(content);

        JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
        JavaFileManager manager = new MemoryFileManager(compiler.getStandardFileManager(null, null, null));

        List<String> options = new ArrayList<String>();
        options.addAll(Arrays.asList("-classpath", System.getProperty("java.class.path")));

        List<JavaFileObject> files = new ArrayList<JavaFileObject>();
        files.add(new MemoryJavaFileObject(name, content));

        compiler.getTask(null, manager, null, options, null, files).call();

        Object instance = manager.getClassLoader(null).loadClass(name).newInstance();
        System.out.println(instance);
    }

}

unfortunately I don't get the expected output but rather:

import lombok.Data;
public @Data class Person {
    private String name;
}

/Person.java:2: warning: Can't initialize javac processor due to (most likely) a class loader     problem: java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment
public @Data class Person {
             ^
      at lombok.javac.apt.Processor.init(Processor.java:84)
      at lombok.core.AnnotationProcessor$JavacDescriptor.want(AnnotationProcessor.java:87)
      at lombok.core.AnnotationProcessor.init(AnnotationProcessor.java:141)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment$ProcessorState.<init>(JavacProcessingEnvironment.java:500)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:597)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:690)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
      at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1173)
      at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:859)
      at com.sun.tools.javac.main.Main.compile(Main.java:523)
      at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
      at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
      at example.Lombok.main(Lombok.java:42)
  Caused by: java.lang.ClassNotFoundException: com.sun.tools.javac.processing.JavacProcessingEnvironment
      at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 15 more
1 warning
Person@39aeed2f

Notice that the class gets compiled and the default toString() method is executed since the typical output is displayed. Also notice that if I run the former example, now I get the following:

public class Person {
    @Override
    public String toString() {
        return "Hello, world!";
    }
}

/Person.java:1: warning: Can't initialize javac processor due to (most likely) a class loader problem: java.lang.NoClassDefFoundError: com/sun/tools/javac/processing/JavacProcessingEnvironment
public class Person {
       ^
      at lombok.javac.apt.Processor.init(Processor.java:84)
      at lombok.core.AnnotationProcessor$JavacDescriptor.want(AnnotationProcessor.java:87)
      at lombok.core.AnnotationProcessor.init(AnnotationProcessor.java:141)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment$ProcessorState.<init>(JavacProcessingEnvironment.java:500)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:597)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:690)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
      at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
      at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1173)
      at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:859)
      at com.sun.tools.javac.main.Main.compile(Main.java:523)
      at com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:129)
      at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:138)
      at example.Simple.main(Simple.java:44)
  Caused by: java.lang.ClassNotFoundException: com.sun.tools.javac.processing.JavacProcessingEnvironment
      at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 15 more
1 warning
Hello, world!

Apparently, by looking at the warning message passed by the exception, lombok doesn't hook the given compiler properly. Unfortunately I was not able to find any useful bit of information. I can only think it could be lombok not dealing properly with Java JDK 8. Am I right?

Do you know any other way to work around this problem?

Inceptive

Are you sure you have a JDK rather than a JRE and the tools.jar is in the class path? – Morly

Hi @Morly and thanks for your reply! Yes, I have Java SDK 8 but I'm on a Mac! And apparently tools.jar is missing on Mac (see #5616818). Thanks for pointing me towards the right direction! – Inceptive

This is weird... I just checked and I have tools.jar in $JAVA_HOME/lib/tools.jar. However, it is not present within Eclipse... It's actually Eclipse's fault: the reference Java Environment is configured as a JRE. Including tools.jar makes everything work. Thanks again! – Inceptive

Maybe you have to add it to the Eclipse project classpath manually… – Morly

Just for reference, in the end I told gradle to add it to the project as a dependency: dependencies { compile files("${System.properties['java.home']}/../lib/tools.jar") compile 'org.projectlombok:lombok:1.14.4' testCompile 'junit:junit:4.11' } – Inceptive

You can (and should) put the solution into an answer and accept that answer so that others can easily see that there’s a solution from the question overview. – Morly

You're right! I couldn't find the Answer button... now I have! Thanks again! – Inceptive

Answer 1

Thanks to Holger, I successfully solved the problem.

The issue was caused by the absence of tools.jar in the class path. This is due to the fact that Eclipse by default recognises the Java environment as a JRE instead of a JDK.

On top of that, Java JDK may - or may not, depending on which version you have - have the tools.jar file.

If you have Java 7 or 8, you should have such library in $JAVA_HOME/lib/tools.jar.

If you have Java 6, the file is not present but the same functionality is provided by $JAVA_HOME/Classes/classes.jar.

The compiler is a feature added with Java 6, so if you want to use it and you have an older version of Java, you should update your environment first.

Now, there are several ways to include tools.jar (or classes.jar) into your project's class path; since I use gradle, I decided to introduce it as a dependency, as you can see in the following snippet of code:

dependencies {
    compile files("${System.properties['java.home']}/../lib/tools.jar")
    compile 'org.projectlombok:lombok:1.14.4'
    testCompile 'junit:junit:4.11'
}

Hope this little explanation might help other people facing a similar problem!

Cheers!

Inceptive

Just in case if something changed within these 3 years: have there been developed a cleaner method to resolve this issue? I have a similar problem on IntelliJ right now, have posted a question referencing yours. – Oe

No idea, I've moved to Python in a meanwhile... nice language, by the way! – Inceptive

In my dreams I am C#ing at work again. Never had such problems there. – Oe

By the way, common patches like adding a dependency didn't work for me. – Oe

What about when we move to java 11 ? we don't have tools.jar there. – Catchment

That was my case, somehow installer did put c:\Program Files\Common Files\Oracle\Java\javapath\java.exe to PATH. Changing to real java path fixed the problem. – Unbroken