How to include all dependent Jars within a single non-executable jar?

Question

I am stuck in a very common problem.

I am plugging my jar (which has many dependencies on third party vendor) into an application server lib directory. If I just copy my jar along with its dependencies into server lib then server classpath becomes to long and hence server is not able to work. Therefore I want to package this Jar with all its dependencies in a single jar so that server's classpath doesn't become too long. I found on various forums that there is a utility to do this i.e. OneJar. But this utility works on executable jar. In my case, my final jar will not be executable.

Also I tried ZIPFileSetGroup utility provided by ANT but that is causing security issues with Manifest file.

Can you please help me in resolving this issue?

Thanks!

Odorous

Are you deploying your app as a WAR file? What app server is reporting the problem? Tomcat, etc? – Vinna

There is a parent JVM (Jetty server) which instantiates child process on request. Now child process picks up classpath from server/lib directory. Since I have added my custom jars into server/lib causing classpath to become too long therefore when parent process tries to instantiate child process using java -cp ... then it fails due to long classpath – Odorous

you need to get a less broken application server. Which one is it? – Rozina

Refer #575094 – Whirlpool

Answer 1

If you use Maven to build, you can use the maven dependency plugin and use the copy-dependency task. It will copy all dependencies into your jar file when it creates it.

If you manually add the jars to your jar file, then you need to make sure your jar file has a Manifest.mf file in it and specify the main class and classpath inside of that.

Manifest-Version: 1.0
Main-Class: com.mypackage.MainClass
Class-Path: my.jar log4j.jar

Another option may be to build an .ear file, that is usually how you see enterprise apps or a .war file for web apps when they package specific jar files with them. It sounds like you are using a server, so one of those formats may be a better fit for you.

Drugge

These jar files are not in maven repository. They are provided by third party vendor. They are many jar files i.e. around 300 jars. Many jars are signed jars. – Odorous

What I did one time was let Maven creates an Executable jar file, then open the jar and see what it put in the Manifest.mf file for the classpath. What error do you get when you put those jars in your lib folder? You said the path was too long? – Drugge

Server is not able to instantiate the child jvm as its classpath becomes too long. The error is as below - – Odorous

INFO | jvm 1 | 2012/12/28 16:17:01 | <REMOTE>>Usage: java [-options] class [args...] INFO | jvm 1 | 2012/12/28 16:17:01 | <REMOTE>> (to execute a class) INFO | jvm 1 | 2012/12/28 16:17:01 | <REMOTE>> or java [-options] -jar jarfile [args...] INFO | jvm 1 | 2012/12/28 16:17:01 | <REMOTE>> (to execute a jar file) INFO | jvm 1 | 2012/12/28 16:17:01 | <REMOTE>> INFO | jvm 1 | 2012 – Odorous

Any idea why it's trying to launch a child jvm? Is this a web service or web application? I've only used 3rd party jars in my deployed application to reference its classes or methods. – Drugge

It is server's functionality to launch child jvm. This is what expected from server but since classpath of child process (-cp /server/lib) becomes too long therefore child process is not being invoked. Server is also a java process (a platform) which is responsible for instantiating child processes. – Odorous

It sounds like with JDK 6, you can include wildcards in the classpath. You may want to check this link out and see if that works for you. unserializableone.blogspot.com/2007/10/… – Drugge

using wildcards might help, need to check this if it can work. – Odorous

I just stumble upon this thread. I know it's too late to comment but did that wild card work? Try putting only one jar file each line and put \ to continue to the next line. – Cacao

Answer 2

Using zipgroupfileset in the jar task in ANT is the easiest approach.

<jar destfile="MyApplication.jar" filesetmanifest="mergewithoutmain">
  <zipgroupfileset dir="lib" includes="*.jar" /> 
  <!-- other options -->
  <manifest>
    <attribute name="Main-Class" value="Main.MainClass" />
  </manifest>
</jar>

Note the filesetmanifest flag set to mergewithoutmain merges everything but the Main section of the manifests.

Signed jars are causing the SecurityException which need to be handled manually. If any classes associated with signed jars verify the signature on the jar as a whole then those will fail at runtime. Digest signatures against a particular file will be added to the manifest without a problem. Since problem is your classpath getting too large you may not be able to bundle all the jars into a single jar but merge most of them making the CLASSPATH manageable.

There is also : http://code.google.com/p/jarjar/

Create target directory with all dependent jars. Next move 10 jars into a temp directory and keep moving the jars in batches of 10 and each time try to create the single jar from that group. When you get the security exception you can isolate which one is causing the problem. Try divide-and-conquer approach. If you have 300 jars then only have to do this 30 times.

Vinna

I already tried zipgroupfileset, this gives Security exception. Also in my case, it is non-executable file i.e. with no main method in it. – Odorous

Did you try with filesetmanifest option? If all else fails jarjar works. – Vinna

@BChawla Was the security exception during construction of the jar, or on the server during attempted use? – Consistory

@PatriciaShanahan, it came during runtime i.e. when server attempted to use it – Odorous

Suggest you remove the signed jars from the list when building single-jar then manually include the classes from those jars. – Vinna

By repacking it you invalidate the signatures. – Diffractometer

It is not easy for me to manually include the classes from those jars. They are many in number. – Odorous

May be time consuming but probably only a handful of signed jars are throwing exceptions. First copy all the jars to a target directory, try to build jar as described then remove any jar that causes a SecurityException. Then bundle your single jar with the remaining jars that were removed in your server app. – Vinna

More than 85 jars have .SF, .DSA or .RSA in Manifest :( – Odorous

You may need to do some scripting to automate part of this process. Perhaps fold jars into the combined jars in groups, splitting a group down if it causes a security exception. – Consistory

Answer 3

When you say

child process picks up classpath from server/lib directory

is this a process that is under your control? If the parent process were to specify the classpath just as

server/lib/*

(i.e. a literal *) then the target java process will enumerate the jar files in the lib directory itself - they do not all need to be named on the classpath.

But if the parent process is explicitly enumerating server/lib/*.jar to build the -cp value then you could take advantage of the fact that the Class-Path in a JAR manifest takes effect even if a JAR is not "executable". You could use a stanza like this to create a manifest-only JAR file

<!-- location of your 300 dependency JAR files, file1.jar ... file300.jar -->
<property name="lib.dir" location="lib" />
<fileset id="dependencies" dir="${lib.dir}" includes="*.jar" />

<pathconvert property="manifest.classpath" dirsep="/" pathsep=" "
             refid="dependencies">
  <map from="${lib.dir}" to="myapp" />
</pathconvert>

<jar destfile="myapp-manifest.jar">
  <manifest>
    <attribute name="Class-Path" value="${manifest.classpath}" />
  </manifest>
</jar>

This will produce a JAR file named myapp-manifest.jar whose manifest contains

Class-Path: myapp/file1.jar myapp/file2.jar ... myapp/file300.jar

You put this file into server/lib and the 300 dependencies into a new directory server/lib/myapp. Now the generated -cp will include just one file (myapp-manifest.jar) but the resulting java process will have all the 300 myapp JAR files available to it.

Sanctimony