How to bundle a native library and a JNI library inside a JAR?

Question

The library in question is Tokyo Cabinet.

I want is to have the native library, JNI library, and all Java API classes in one JAR file to avoid redistribution headaches.

There seems to be an attempt at this at GitHub, but

1. It does not include the actual native library, only JNI library.
2. It seems to be specific to Leiningen's native dependencies plugin (it won't work as a redistributable).

The question is, can I bundle everything in one JAR and redistribute it? If yes, how?

P.S.: Yes, I realize it may have portability implications.

Frit

Answer 1

It is possible to create a single JAR file with all dependencies including the native JNI libraries for one or more platforms. The basic mechanism is to use System.load(File) to load the library instead of the typical System.loadLibrary(String) which searches the java.library.path system property. This method makes installation much simpler as the user does not have to install the JNI library on his system, at the expense, however, that all platforms might not be supported as the specific library for a platform might not be included in the single JAR file.

The process is as follows:

• include the native JNI libraries in the JAR file at a location specific to the platform, for example at NATIVE/${os.arch}/${os.name}/libname.lib
• create code in a static initializier of the main class to
  • calc the current os.arch and os.name
  • look for the library in the JAR file at the predefined location using Class.getResource(String)
  • if it exists, extract it to a temp file and load it with System.load(File).

I added functionality to do this for jzmq, the Java bindings of ZeroMQ (shameless plug). The code can be found here. The jzmq code uses a hybrid solution so that if an embedded library cannot be loaded, the code will revert to searching for the JNI library along the java.library.path.

Suspensoid

I love it! It saves a lot of trouble for integration, and you can always revert to the "old" way with System.loadLibrary() in case it failed. I think I'll start using that. Thanks! :) – Involutional

What do I do if my library dll has a dependency to another dll? I always get an UnsatisfiedLinkError as loading the former dll implicitly wants to load the latter, but cannot find it as it is hidden in the jar. Also loading the latter dll first does not help. – Giuseppinagiustina

The other dependents DLL must be known in advance and their locations should be added in the PATH environment variable. – Bailee

Works great! If it's not clear to someone coming across this, drop the EmbeddedLibraryTools class in to your project and change it up accordingly. – Tincture

@JonLaMarr Only go that route if your code is GPL licensed, as jzmq is. – Angelaangele

Answer 2

https://www.adamheinrich.com/blog/2012/12/how-to-load-native-jni-library-from-jar/

is great article, which solves my issue ..

In my case I've got the following code for initialize the library:

static {
    try {
        System.loadLibrary("crypt"); // used for tests. This library in classpath only
    } catch (UnsatisfiedLinkError e) {
        try {
            NativeUtils.loadLibraryFromJar("/natives/crypt.dll"); // during runtime. .DLL within .JAR
        } catch (IOException e1) {
            throw new RuntimeException(e1);
        }
    }
}

Cheshvan

use NativeUtils.loadLibraryFromJar("/natives/"+System.mapLibraryName("crypt")); could be better – Brockman

Hi when I am trying to use the NativeUtils class and trying to lod the .so file inside the libs/armeabi/libmyname.so, i am getting the exception like the java.lang.ExceptionInInitializerError, Caused by: java.io.FileNotFoundException: File /native/libhellojni.so was not found inside JAR. Please let me know why I am getting the exception. Thank you – Devote

Answer 3

Take a look at One-JAR. It will wrap your application up in a single jar file with a specialised class loader which handles "jars within jars" among other things.

It handles native (JNI) libraries by unpacking them to a temporary working folder as required.

(Disclaimer: I've never used One-JAR, haven't needed to as yet, just had it bookmarked for a rainy day.)

Wherefrom

I'm not a Java programmer, any idea if I have to wrap the application itself? How this would work in combination with an existing classloader? To clarify, I'm going to use it from Clojure and want to be able to load a JAR as a library, rather than as an application. – Frit

Ahhh, it probably wouldn't be suitable than. Think you'll be stuck with distributing your native library(s) outside of the jar file, with instructions to put them in the application's library path. – Wherefrom

Answer 4

1) Include the native library into your JAR as a Resource. E. g. with Maven or Gradle, and the standard project layout, put the native library into main/resources directory.

2) Somewhere in static initializers of Java classes, related to this library, put the code like the following:

String libName = "myNativeLib.so"; // The name of the file in resources/ dir
URL url = MyClass.class.getResource("/" + libName);
File tmpDir = Files.createTempDirectory("my-native-lib").toFile();
tmpDir.deleteOnExit();
File nativeLibTmpFile = new File(tmpDir, libName);
nativeLibTmpFile.deleteOnExit();
try (InputStream in = url.openStream()) {
    Files.copy(in, nativeLibTmpFile.toPath());
}
System.load(nativeLibTmpFile.getAbsolutePath());

Guanabara

This solution also works in case you want to deploy something to an application server like wildfly AND the best part is that it is able to unload the application properly! – Cauvery

This is the best solution to avoid 'native library already' loaded exception. – Loux

Answer 5

JarClassLoader is a class loader to load classes, native libraries and resources from a single monster JAR and from JARs inside the monster JAR.

Fatten

Answer 6

Solution for Kotlin:

• build.gradle.dsl: copy kotlin runtime (kotlin-stdlib-1.4.0.jar) and native library (librust_kotlin.dylib) to JAR

  tasks.withType<Jar> {
      manifest {
          attributes("Main-Class" to "MainKt")
      }
  
      val libs = setOf("kotlin-stdlib-1.4.0.jar")
  
      from(configurations.runtimeClasspath.get()
          .filter { it.name in libs }
          .map { zipTree(it) })
  
      from("librust_kotlin.dylib")
  }
  

• main method: copy library to a temporary file to load it using absolute path

   with(createTempFile()) {
       deleteOnExit()
       val bytes = My::class.java.getResource("librust_kotlin.dylib")
           .readBytes()
  
       outputStream().write(bytes)
       System.load(path)
   }
  

Bootle

Answer 7

You will probably have to unjar the native library to the local file system. As far as I know the bit of code that does the native loading looks at the file system.

This code should help get you started (I haven't looked at it in a while, and it is for a different purpose but should do the trick, and I am pretty busy at the moment, but if you have questions just leave a comment and I'll answer as soon as I can).

import java.io.Closeable;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.net.URLDecoder;
import java.security.CodeSource;
import java.security.ProtectionDomain;
import java.util.zip.ZipEntry;
import java.util.zip.ZipException;
import java.util.zip.ZipFile;


public class FileUtils
{
    public static String getFileName(final Class<?>  owner,
                                     final String    name)
        throws URISyntaxException,
               ZipException,
               IOException
    {
        String    fileName;
        final URI uri;

        try
        {
            final String external;
            final String decoded;
            final int    pos;

            uri      = getResourceAsURI(owner.getPackage().getName().replaceAll("\\.", "/") + "/" + name, owner);
            external = uri.toURL().toExternalForm();
            decoded  = external; // URLDecoder.decode(external, "UTF-8");
            pos      = decoded.indexOf(":/");
            fileName = decoded.substring(pos + 1);
        }
        catch(final FileNotFoundException ex)
        {
            fileName = null;
        }

        if(fileName == null || !(new File(fileName).exists()))
        {
            fileName = getFileNameX(owner, name);
        }

        return (fileName);
    }

    private static String getFileNameX(final Class<?> clazz, final String name)
        throws UnsupportedEncodingException
    {
        final URL    url;
        final String fileName;

        url = clazz.getResource(name);

        if(url == null)
        {
            fileName = name;
        }
        else
        {
            final String decoded;
            final int    pos;

            decoded  = URLDecoder.decode(url.toExternalForm(), "UTF-8");
            pos      = decoded.indexOf(":/");
            fileName = decoded.substring(pos + 1);
        }

        return (fileName);
    }

    private static URI getResourceAsURI(final String    resourceName,
                                       final Class<?> clazz)
        throws URISyntaxException,
               ZipException,
               IOException
    {
        final URI uri;
        final URI resourceURI;

        uri         = getJarURI(clazz);
        resourceURI = getFile(uri, resourceName);

        return (resourceURI);
    }

    private static URI getJarURI(final Class<?> clazz)
        throws URISyntaxException
    {
        final ProtectionDomain domain;
        final CodeSource       source;
        final URL              url;
        final URI              uri;

        domain = clazz.getProtectionDomain();
        source = domain.getCodeSource();
        url    = source.getLocation();
        uri    = url.toURI();

        return (uri);
    }

    private static URI getFile(final URI    where,
                               final String fileName)
        throws ZipException,
               IOException
    {
        final File location;
        final URI  fileURI;

        location = new File(where);

        // not in a JAR, just return the path on disk
        if(location.isDirectory())
        {
            fileURI = URI.create(where.toString() + fileName);
        }
        else
        {
            final ZipFile zipFile;

            zipFile = new ZipFile(location);

            try
            {
                fileURI = extract(zipFile, fileName);
            }
            finally
            {
                zipFile.close();
            }
        }

        return (fileURI);
    }

    private static URI extract(final ZipFile zipFile,
                               final String  fileName)
        throws IOException
    {
        final File         tempFile;
        final ZipEntry     entry;
        final InputStream  zipStream;
        OutputStream       fileStream;

        tempFile = File.createTempFile(fileName.replace("/", ""), Long.toString(System.currentTimeMillis()));
        tempFile.deleteOnExit();
        entry    = zipFile.getEntry(fileName);

        if(entry == null)
        {
            throw new FileNotFoundException("cannot find file: " + fileName + " in archive: " + zipFile.getName());
        }

        zipStream  = zipFile.getInputStream(entry);
        fileStream = null;

        try
        {
            final byte[] buf;
            int          i;

            fileStream = new FileOutputStream(tempFile);
            buf        = new byte[1024];
            i          = 0;

            while((i = zipStream.read(buf)) != -1)
            {
                fileStream.write(buf, 0, i);
            }
        }
        finally
        {
            close(zipStream);
            close(fileStream);
        }

        return (tempFile.toURI());
    }

    private static void close(final Closeable stream)
    {
        if(stream != null)
        {
            try
            {
                stream.close();
            }
            catch(final IOException ex)
            {
                ex.printStackTrace();
            }
        }
    }
}

Saturday

If you need to unjar some specific resource, I recommend to have a look at github.com/zeroturnaround/zt-zip project – Walczak

zt-zip looks like a decent API. – Saturday