web start jar validation getting slower with each Java update

Question

We have an (Eclipse RCP) application of 90MB with 139 self-signed jars which starts in 8s without Web Start and in 10s in an older version of Java 7. We configured Java to not use the browser proxy, i.e. deployment.proxy.type=0.

With each update of Oracle's Java startup performance drops. It takes more and more time to fully start up:

• 7u60/7u65/8u25: 13s (starts after 5s of web start processing)
• 7u75: 23s
• 8u31: 20s
• 8u40: 29s
• 8u51/8u60ea: 32s

What can I do to solve this issue?

From the trace/logs I can see that it is very probable that this slowdown is completely due to validating the cached jars taking much more time. Note that this question is similar but doesn't provide the following details:

Diagnosis:

• When cached, the update check runs in only 0.5s (server returns "304 Not Modified"), but even with a full download it takes only a few seconds on the gigabit network.
• After the update check, for each jar XXX there is a log entry:

validating cached jar XXX.jar

When this is done, com.sun.javaws.Main is started after which the same validation seems to happen again and takes about the same amount of time, then the application starts.

• The time spent in validating the cached jars seems to correspond to the extra time required before the application starts.
• The web start splash screen always shows for about 2 seconds corresponding to the update check and is then hidden. Then after almost 20 seconds the Java console finally appears and my application actually starts.
• During the delay, jp2launcher.exe uses about 16% processor time on a quad core with hyperthreading (8 logical cores). So it looks like it is fully using one of the logical processors.

What I have already tried but did not make any difference:

• clearing web start cache (countless times)
• configure deployment.properties to disable certificate revocation check (as well as blacklist.check and validation.ocsp, validation.crl)
• running offline
• using the version download protocol
• add to site exceptions list
• check web server logs for problems. None found, update check runs in about 500ms for all 138 jars.
• use another web server
• checked certificate expiration date 17 feb 2016
• validated my jnlp with JaNela and found no serious issues
• create a deployment rule set to allow the application to run unsigned in order to speed up validation. This should be possible and looks like a promising way to solve this, but I could not get it working. See also my answer on this post.
• configured Java to "do not start the console"

Detail: some weird behavior on 7u60 In 7u60 the application is started after about 5 seconds, after which the Java console APPEARS to be doing the jar validation in the background while the application is already started. HOWEVER the .log file reports that the application gets started AFTER all the validations are done. It reports this as 25 seconds and then shows the first System.out of my application which actually happened after only 5 seconds or so. It also shows the jar update check with the server taking 10 times as long as reported by the server. So I guess this is an issue with the logging framework lagging behind! Haven't seen this on 8u51.

Spinode

When I run "jarsigner -verify" on all of the jars, it takes about 30 seconds, even with 7u60. In principle no conclusions can be drawn from this becomes this time includes the overhead of starting a new jarsigner process for each jar. But it does lead me to believe that verifying jars has always been very slow (like signing them) and that older Java Web Start versions had some kind of optimization that is removed in newer versions. Perhaps they cached validation results or perhaps they lazily validated jars as they were loaded instead of eagerly validating them all. – Spinode

I decided to implement my own "LAN Start" mechanism. In fact it's just a short batch script which copies the client through a local network share if the version.properties file on the client and server do not match, and then launch the application. Very simple and very fast, but of course only works in local network. – Spinode

Answer 1

Not an answer per se (yet), but I found that Java 8u25 when tracing is enabled, only generate a single trace file. 8u51 generates two files, one from the JVM used to update the application and other to run it. This is new (two JVM startups) and I think is related to the new setting for using native Windows sandbox capabilities. The problem is that it shouldn't have to validate the signatures again on the second JVM. The separation on two JVM instances always happen no matter if the setting for using native sandbox is disabled (the default).

I reported a regression bug, will edit the answer if I get an answer from Oracle.

Note: Java 8u31 still runs everything on one JVM but have the same doubled startup time the question stated.

Winter

Indeed I found that 7u60/8u25 seem to run the application in the same JVM through the same javaws.exe process. Later versions launch it through a separate jp2launcher.exe process. This already happens in 7u65 which does not give the performance problem. The jp2launcher could indeed explain why I am seeing the jars being validated twice. Even if this affects performance it could not explain the degradation from 13s to 32s: with 7u60, Web start takes an extra 5s on top of the 8s to launch the application, so it should result in at most 2 * 5s + 8s = 18s. – Spinode