"java.io.IOException: Invalid secret key format" when opening JCEKS key store with Oracle Java 8 JRE 172

Question

I am getting the following exception when I try to open JCEKS type key store with with Oracle Java 8 JRE 172 on Windows. This worked fine with earlier versions of the JRE:

INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 1, depth: 1, bytes: 70, ex: n/a
[...call stacks omitted to protect the innocent...]
Caused by: java.io.IOException: Invalid secret key format
        at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:856)
        at java.security.KeyStore.load(Unknown Source)
[...]

This looks very much like JDK-8202506 but I use Java 8 and I get null in the initial INFO message.

Is this the same issue?

It seems to me the JDK-8202506 issue is currently not fixed in any public JRE release. Am I right?

UPDATE 1

This looks similar and they also have no solution: ATLAS-2642

UPDATE 2

For some reason, Equinox fails to see the com.sun.crypto.provider.SealedObjectForKeyProtector class after the upgrade, even though it is clearly in the JRE that comes with the new JDK:

BundleClassLoader[foo.bar.baz.crypto].loadClass(com.sun.crypto.provider.SealedObjectForKeyProtector) failed.
java.lang.ClassNotFoundException: com.sun.crypto.provider.SealedObjectForKeyProtector
    at org.eclipse.osgi.framework.internal.core.BundleLoader.findClassInternal(BundleLoader.java:481)
    at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:397)
    at org.eclipse.osgi.framework.internal.core.BundleLoader.findClass(BundleLoader.java:385)
    at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:87)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:686)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:850)
    at java.security.KeyStore.load(KeyStore.java:1445)

UPDATE 3

The class SealedObjectForKeyProtector.class is somehow different from the rest of the classes in the sunjce_provider.jar. When we try to decompile it with JD-GUI, it fails with internal error, unlike the rest of the classes:

Abnaki

Answer 1

I meet this issue these days. And per my troubleshooting, it is caused by the different return value of this method:

sun.misc.VM.latestUserDefinedLoader()

Previously (before 8u171), this method returns sun.misc.Launcher$ExtClassLoader, while it returns application's classloader after upgrade. In ObjectInputStream, both class loader can load com.sun.crypto.provider.SealedObjectForKeyProtector successfully, that's simply because ExtClassLoader is the parent of application's class loader (or, parent's parent). However, once SealedObjectForKeyProtector is loaded by application's class loader, it's class loader no longer equals to ExtClassLoader.

On the other hand, within com.sun.crypto.provider.JceKeyStore, unlike ObjectInputStream, SealedObjectForKeyProtector is always loaded by ExtClassLoader. Thus below check in JceKeyStore.java:932 will fail due to class doesn't equal:

932            if (info.serialClass() != SealedObjectForKeyProtector.class))
934                return Status.REJECTED;

Then, we will get below log and an IOException eventually: ObjectInputFilter REJECTED: class com.sun.crypto.provider.SealedObjectForKeyProtector

Solution: make sure class com.sun.crypto.provider.SealedObjectForKeyProtector is not loaded by ContextClassLoader by certain configuration. Details depend on the ContextClassLoader. For example, for org.powermock.core.classloader.MockClassLoader, the concrete solution is adding below annotation to involved test classes:

@PowerMockIgnore("com.sun.*")

Gumshoe

Answer 2

I have done a complete analysis about the issue and debugged through the code part that involves in JCEKS keystore. Whenever the application uses custom class loader then you will face the issue for sure, if using JCEKS keystore from older java versions than JDK 8 Update 151.

 private static class DeserializationChecker implements ObjectInputFilter {
    private static final int MAX_NESTED_DEPTH = 2;

    @Override
    public ObjectInputFilter.Status
        checkInput(ObjectInputFilter.FilterInfo info) {

        // First run a custom filter
        long nestedDepth = info.depth();
        if ((nestedDepth == 1 &&
                    info.serialClass() != SealedObjectForKeyProtector.class) ||
                (nestedDepth > MAX_NESTED_DEPTH &&
                    info.serialClass() != null &&
                    info.serialClass() != Object.class)) {
            return Status.REJECTED;
        }

        // Next run the default filter, if available
        ObjectInputFilter defaultFilter =
            ObjectInputFilter.Config.getSerialFilter();
        if (defaultFilter != null) {
            return defaultFilter.checkInput(info);
        }

        return Status.UNDECIDED;
    }
}

In the above code from JceKeyStore.class, the filter info will always be null, so info.serialClass() != SealedObjectForKeyProtector.class check will fail. The class loading delegation doesnot happening as expected in case of custom class loader like Equinox - Eclipse plugin.

There is a two step solution

1. Firstly upgrade the keystore from JCEKS to PKCS12 as mentioned in the JDK 151 https://www.oracle.com/technetwork/java/javase/8u151-relnotes-3850493.html under Notes: Better keystore handling topic. For most cases this will solve the issue. Further access the keystore with keystore type value PKCS12 after changes.
2. Make the classes from sun_jce provider get loaded earlier even before your custom class loaders scope. Use those configuration to make your custom class loader delegate the class loading of sun_jce jars to ExtClassLoader. For example : By using Eclipse-BuddyPolicy: ext in the META-INF of eclipse plugin projects.

Unaccountedfor

We have ended up re-implementing our keystore with PKCS12 format. I think, it has been a while now, I tried Eclipse-BuddyPolicy: ext and it did not seem to help but I cannot be completely sure. – Abnaki

Answer 3

I am currently working with Oracle JRE Support on this and have a private bug opened. Information I got so far:

1. It's not the same as JDK-8202506.
2. Yes it's a classloading issue on SealedObjectForKeyProtector with Equinox.

Solutions:

A workaround is to add the following line into the OSGi bundle MANIFEST.MF.

Eclipse-BuddyPolicy: ext

I personally verified this workaround with JRE 1.8_181 and it seems working.

I was also told that for Java 9, a JVM parameter -Dosgi.compatibility.bootdelegation=true can do the job (w/o the need to update MANIFEST.MF) but I don't have a Java 9 environment to verify it. Appreciate if someone can try it and let us know the result.

Borlase

Well, I am eagerly waiting for a resolution. :) – Abnaki

I added my answer with a verified solution. – Borlase

I am not sure this is entirely correct. It did not seem to have an effect for me. – Abnaki

@Abnaki Sorry to know. :-( I am wondering if you may want to share your test case with me via some channels such as google drive or Dropbox etc. – Borlase