Scanning classpath/modulepath in runtime in Java 9
Asked Answered
H

2

24

I can't seem to find any info on whether scanning all available classes (for interfaces, annotations etc) is still possible in runtime, the way Spring, Reflections and many other frameworks and libraries currently do, in the face of Jigsaw related changes to the way classes are loaded.

EDIT: This question is about scanning the real physical file paths looking for classes. The other question is about dynamically loading classes and resources. It's related but very much not a duplicate.

UPDATE: Jetty project has made a JEP proposal for a standardized API for this. If you have a way to help make this reality, please do. Otherwise, wait and hope.

UPDATE 2: Found this relevant sounding post. Quoting the code snippet for posterity:

If you are really just looking to get at the contents of the modules in the boot layer (the modules that are resolved at startup) then you'll do something like this:

  ModuleLayer.boot().configuration().modules().stream()
         .map(ResolvedModule::reference)
         .forEach(mref -> {
             System.out.println(mref.descriptor().name());
             try (ModuleReader reader = mref.open()) {
                 reader.list().forEach(System.out::println);
            } catch (IOException ioe) {
                 throw new UncheckedIOException(ioe);
             }
         });
Headlock answered 30/1, 2017 at 9:36 Comment(6)
I don't think much has changed, I mean if there is visibility of one module towards another, you can still do the usual things. In case you can't access a class from a module you would probably hit an exception like InaccessibleObjectException or something like thatKurrajong
@Kurrajong You sure? Asking because classpath now seems to be a "legacy" thing, getting replaced by modulepath. So I'm guessing scanning what used to be classpath has changed...Headlock
Possible duplicate of Loading classes and resources in Java 9Yeasty
@MichaelEaster It is related, but not a duplicate. This one is about scanning directories and URLs for available classes, that one is about loading them.Headlock
See my answer below for scanning more than just the boot module layer.Cereal
github.com/classgraph/classgraph/wiki/Code-examples The ClassGraph (previously FastClassGraph) has been helpful for me.Cargo
C
30

The following code achieves module path scanning in Java 9+ (Jigsaw / JPMS). It finds all classes on the callstack, then for each class reference, calls classRef.getModule().getLayer().getConfiguration().modules(), which returns a a List<ResolvedModule>, rather than just a List<Module>. (ResolvedModule gives you access to the module resources, whereas Module does not.) Given a ResolvedModule reference for each module, you can call the .reference() method to get the ModuleReference for a module. ModuleReference#open() gives you a ModuleReader, which allows you to list the resources in a module, using ModuleReader#list(), or to open a resource using Optional<InputStream> ModuleReader#open(resourcePath) or Optional<ByteBuffer> ModuleReader#read(resourcePath). You then close the ModuleReader when you're done with the module. This is not documented anywhere that I have seen. It was very difficult to figure all this out. But here is the code, in the hope that someone else will benefit from this.

Note that even in JDK9+, you can still utilize traditional classpath elements along with module path elements, so for a complete module path + classpath scan, you should probably use a proper classpath scanning solution, such as ClassGraph, which supports module scanning using the below mechanism (disclaimer, I am the author). You can find a reflection-based version of the following code here.

Also note that there was a bug in StackWalker in several JDK releases after JDK 9 that has to be worked around, see the above reflection-based code for details.

package main;

import java.lang.StackWalker;
import java.lang.StackWalker.Option;
import java.lang.StackWalker.StackFrame;
import java.lang.module.ModuleReader;
import java.lang.module.ModuleReference;
import java.lang.module.ResolvedModule;
import java.net.URI;
import java.security.AccessController;
import java.security.PrivilegedAction;
import java.util.AbstractMap.SimpleEntry;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Deque;
import java.util.HashSet;
import java.util.List;
import java.util.Map.Entry;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Stream;

public class Java9Scanner {

    /** Recursively find the topological sort order of ancestral layers. */
    private static void findLayerOrder(ModuleLayer layer,
            Set<ModuleLayer> visited, Deque<ModuleLayer> layersOut) {
        if (visited.add(layer)) {
            List<ModuleLayer> parents = layer.parents();
            for (int i = 0; i < parents.size(); i++) {
                findLayerOrder(parents.get(i), visited, layersOut);
            }
            layersOut.push(layer);
        }
    }

    /** Get ModuleReferences from a Class reference. */
    private static List<Entry<ModuleReference, ModuleLayer>> findModuleRefs(
            Class<?>[] callStack) {
        Deque<ModuleLayer> layerOrder = new ArrayDeque<>();
        Set<ModuleLayer> visited = new HashSet<>();
        for (int i = 0; i < callStack.length; i++) {
            ModuleLayer layer = callStack[i].getModule().getLayer();
            findLayerOrder(layer, visited, layerOrder);
        }
        Set<ModuleReference> addedModules = new HashSet<>();
        List<Entry<ModuleReference, ModuleLayer>> moduleRefs = new ArrayList<>();
        for (ModuleLayer layer : layerOrder) {
            Set<ResolvedModule> modulesInLayerSet = layer.configuration()
                    .modules();
            final List<Entry<ModuleReference, ModuleLayer>> modulesInLayer =
                    new ArrayList<>();
            for (ResolvedModule module : modulesInLayerSet) {
                modulesInLayer
                        .add(new SimpleEntry<>(module.reference(), layer));
            }
            // Sort modules in layer by name for consistency
            Collections.sort(modulesInLayer,
                    (e1, e2) -> e1.getKey().descriptor().name()
                            .compareTo(e2.getKey().descriptor().name()));
            // To be safe, dedup ModuleReferences, in case a module occurs in multiple
            // layers and reuses its ModuleReference (no idea if this can happen)
            for (Entry<ModuleReference, ModuleLayer> m : modulesInLayer) {
                if (addedModules.add(m.getKey())) {
                    moduleRefs.add(m);
                }
            }
        }
        return moduleRefs;
    }

    /** Get the classes in the call stack. */
    private static Class<?>[] getCallStack() {
        // Try StackWalker (JDK 9+)
        PrivilegedAction<Class<?>[]> stackWalkerAction =
                (PrivilegedAction<Class<?>[]>) () ->
                    StackWalker.getInstance(
                            Option.RETAIN_CLASS_REFERENCE)
                    .walk(s -> s.map(
                            StackFrame::getDeclaringClass)
                            .toArray(Class[]::new));
        try {
            // Try with doPrivileged()
            return AccessController
                    .doPrivileged(stackWalkerAction);
        } catch (Exception e) {
        }
        try {
            // Try without doPrivileged()
            return stackWalkerAction.run();
        } catch (Exception e) {
        }

        // Try SecurityManager
        PrivilegedAction<Class<?>[]> callerResolverAction = 
                (PrivilegedAction<Class<?>[]>) () ->
                    new SecurityManager() {
                        @Override
                        public Class<?>[] getClassContext() {
                            return super.getClassContext();
                        }
                    }.getClassContext();
        try {
            // Try with doPrivileged()
            return AccessController
                    .doPrivileged(callerResolverAction);
        } catch (Exception e) {
        }
        try {
            // Try without doPrivileged()
            return callerResolverAction.run();
        } catch (Exception e) {
        }

        // As a fallback, use getStackTrace() to try to get the call stack
        try {
            throw new Exception();
        } catch (final Exception e) {
            final List<Class<?>> classes = new ArrayList<>();
            for (final StackTraceElement elt : e.getStackTrace()) {
                try {
                    classes.add(Class.forName(elt.getClassName()));
                } catch (final Throwable e2) {
                    // Ignore
                }
            }
            if (classes.size() > 0) {
                return classes.toArray(new Class<?>[0]);
            } else {
                // Last-ditch effort -- include just this class
                return new Class<?>[] { Java9Scanner.class };
            }
        }
    }

    /**
     * Return true if the given module name is a system module.
     * There can be system modules in layers above the boot layer.
     */
    private static boolean isSystemModule(
            final ModuleReference moduleReference) {
        String name = moduleReference.descriptor().name();
        if (name == null) {
            return false;
        }
        return name.startsWith("java.") || name.startsWith("jdk.")
            || name.startsWith("javafx.") || name.startsWith("oracle.");
    }

    public static void main(String[] args) throws Exception {
        // Get ModuleReferences for modules of all classes in call stack,
        List<Entry<ModuleReference, ModuleLayer>> systemModuleRefs = new ArrayList<>();
        List<Entry<ModuleReference, ModuleLayer>> nonSystemModuleRefs = new ArrayList<>();

        Class<?>[] callStack = getCallStack();
        List<Entry<ModuleReference, ModuleLayer>> moduleRefs = findModuleRefs(
                callStack);
        // Split module refs into system and non-system modules based on module name
        for (Entry<ModuleReference, ModuleLayer> m : moduleRefs) {
            (isSystemModule(m.getKey()) ? systemModuleRefs
                    : nonSystemModuleRefs).add(m);
        }

        // List system modules
        System.out.println("\nSYSTEM MODULES:\n");
        for (Entry<ModuleReference, ModuleLayer> e : systemModuleRefs) {
            ModuleReference ref = e.getKey();
            System.out.println("  " + ref.descriptor().name());
        }

        // Show info for non-system modules
        System.out.println("\nNON-SYSTEM MODULES:");
        for (Entry<ModuleReference, ModuleLayer> e : nonSystemModuleRefs) {
            ModuleReference ref = e.getKey();
            ModuleLayer layer = e.getValue();
            System.out.println("\n  " + ref.descriptor().name());
            System.out.println(
                    "    Version: " + ref.descriptor().toNameAndVersion());
            System.out.println(
                    "    Packages: " + ref.descriptor().packages());
            System.out.println("    ClassLoader: "
                    + layer.findLoader(ref.descriptor().name()));
            Optional<URI> location = ref.location();
            if (location.isPresent()) {
                System.out.println("    Location: " + location.get());
            }
            try (ModuleReader moduleReader = ref.open()) {
                Stream<String> stream = moduleReader.list();
                stream.forEach(s -> System.out.println("      File: " + s));
            }
        }
    }
}
Cereal answered 10/8, 2017 at 11:24 Comment(12)
Here are more detailed investigations by the same author: github.com/lukehutch/fast-classpath-scanner/issues/36Ovalle
@Headlock I updated my answer with the complete scanning code.Cereal
@LukeHutchison You sir, are the god-emperor of human kind :) Does this mean FastClasspathScanner is now able to deal with Java 9 with modulepath (and no classpath)?Headlock
isSystemModule seems problematic, esp with current EE classes (and the future ee4j jakarta namespace too)Succession
@JoakimErdfelt thanks for pointing this out. Can you give a few examples, please? Do you know any other way of detecting system modules?Cereal
@LukeHutchison perhaps if the location url for the module has scheme jrt:? That means the module belongs to the java-runtime, right?Succession
@JoakimErdfelt actually I just discovered that for a jlink'd project, all URLs become jrt: URLs. I changed isSystemModule() back to use name-based criteria -- specifically the following are assumed to be system modules: java.*, jdk.*, javafx.*, and oracle.*. I left off javafx.*. Does that create a problem with any of the EE modules you were referring to?Cereal
@LukeHutchison the entire Java EE namespace has moved to Eclipse. Which has rebranded as Jakarta EE, already a few modules on maven central with this new namespace - central.maven.org/maven2/jakartaSuccession
@JoakimErdfelt thanks, that's helpful. But are you saying that all of jakarta.* should be treated as system modules? And are the module names the same as the package names?Cereal
You can simplify the stack walker usage to a single return AccessController.doPrivileged((PrivilegedAction<Class<?>[]>)() -> StackWalker.getInstance(Option.RETAIN_CLASS_REFERENCE) .walk(s -> s.map(StackFrame::getDeclaringClass).toArray(Class<?>[]::new))); statement. No need to add to an ArrayList manually nor converting it to an array afterwards. And in Java 9 code, you can use lambda expressions (as you already do at some places), instead of anonymous inner classes.Fetlock
@Fetlock thanks for the improvements! Except that your code doesn't compile unless you replace Class<?>[]::new with Class[]::new due to Type mismatch: cannot convert from Class<?>[] to Class<capture#7-of ?>[]. I'll update my code example though.Cereal
When you still use the variable, you don’t need the type cast. In my example, I passed the lambda directly to doPrivileged, which requires the type cast for disambiguation. But with a local variable, its declared type does already the job. Don’t know why your compiler didn’t accept .toArray(Class<?>[]::new). It should, and all versions of javac, I tried, did.Fetlock
V
2

The actual issue here is to find the paths to all jars and folders on the classpath. Once when you have them, you can scan.

What I did is the following:

  • get the current module descriptor for current class
  • get all requires modules
  • for each such module open resource of MANIFEST.MF
  • remove the MANIFEST.MF path from the resource url
  • what remains is the classpath of the module, i.e. to it's jar or folder.

I do the same for current module, to get the classpath for current code.

This way I collect classpath of a currently working module and all its required modules (1 step away). That was working for me - and my Java8 scanner was still being able to do the job. This approach does not require any additional VM flag etc.

I could extend this approach to get all required modules easily (not only the first level), but for now, I don't need that.

Code.

Voyeurism answered 6/12, 2017 at 11:48 Comment(1)
The module jars on the modulepath are not listed on the traditional classpath in Java 9+.Cereal

© 2022 - 2024 — McMap. All rights reserved.