Why do I get a handshake failure (Java SSL)

Question

I'm connecting to a web service over HTTPS. I've done all that I think is required to make it work, but in the end I get a handshake failure.

I found out that as a new user I can't post more than 2 links due to "spam protection" - thanx a lot stackoverflow...anyway here's a link to a pastebin post with all the links spelled out...so when I write "link#1" here it's a reference to these links: http://pastebin.com/y4zGNRC7

• I verified the same behavior using HttpClient (GET on the service URL) and actually calling the web service via a CXF proxy
• I'm setting both the keystore and truststore - I tried both the "in code" way ( link#1 ) and setting the system properties - i.e. System.setProperty("javax.net.ssl.keyStore", "mykeystore.jks");
• SSL debug is on ( javax.net.debug=all )
• SSL debug blurts out the contents of both keystore and truststore (i.e. looks like java "knows about them") - link#2
• seems like there's some client-server communication going on, but then it crashes for some reason link#3
• I successfully connected to the server using the client and CA certificates both in a browser (Chrome) and using openssl s_client
• wireshark shows less client-server talk from java ( link#4 ) then for example from Chrome ( link#5 )

Another strange thing is, that I seem to be getting the same behavior when I set the keystore and when I don't (the only difference is that when I do the keystore contents get printed in the console, but that's it).

I tried googling the problem and I saw numerous similar posts here on stackoverflow, but nothing helped. I tried changing the protocol version ("TLSv1", "SSLv3", even the weird v2 Hello). Any help would be appreciated - maybe there's some fundamental thing I might have overlooked...I'm getting desperate here... Thanx

PS I'm running java 1.6 update 30 on Fedora Core 15 (64bit)

Hine

Can you post an exception stacktrace? – Balmung

"I found out that as a new user I can't post more than 2 links due to "spam protection" - thanx a lot stackoverflow". If you use this site a lot, you will realize that SO's spam protection measures are a GOOD THING. – Ezekielezell

And on the subject of links, it is considered to be bad form to post pastebin links and similar because they quickly evaporate making the question unintelligible to future readers. – Ezekielezell

Here's the stacktrace pastebin.com/dbXPYRmS – Chum

Yeah sorry I got a little agitated, but I understand how it helps against spam... External links are imo a nice way to "not pollute" the question with too much detail....plus isn't there a character limit? Pastebin lets you choose the "expiration" of your posts...and "never" seems like a good option ;-) It would be nice if stackoverflow itself provided such a function. – Chum

Anyone? I ran out of reasonable things to try... – Chum

Why don't you post the full output of SSL debug? – Moke

Here it is: pastebin.com/K9H3Lqzn – Chum

I think there's a problem with the keystore...I set it and java seems to be aware of it, but then it apparently doesn't send the client certificate to the server. – Chum

Also sometimes (as seen in the debug output), I set the keystore and truststore, those get printed in the console and then the same thing gets printed again except for the fact that the keystore seems empty and the truststore is the default jdk store. But that didn't seem to happen before/every time. I put the CAs to the jdk store too in the process so that wouldn't matter but maybe the keystore settings get overwritten somehow... – Chum

It seems that the server requests a certificate with the CN of the root CA("Cert Authorities") which is strange. I found a page where the SSL debug is nicely commented docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/… . Either the certificate should be one issued by the root CA or having the CN=RootCA. I don't know if this is a bug on the server side or what. The keystore contains the client key, but that is signed/issued by another CA - the "SubCA". – Chum

As I mentioned Chrome and openssl only require the client cert and the SubCA cert to work...strange. – Chum

I tried to make a wrapper for the KeyManager and managed to make it supply the client cert, however the handshake still fails. I assume this is because the whole certificate chain is not supplied (i.e. the server expects to get the client cert and the issuer cert?). That's what Chrome does anyway. I'm puzzled by this behavior. Could this be an improperly configured server? – Chum

Answer 1

The problem was that even though the keystore and truststore was set, java decided not to send the client certificate to the server. The reason for this was the fact, that the server requested a certificate signed by the RootCA authority, but the client certificate is signed by a SubCA authority (which is issued by the RootCA).

Originally the keystore only contained the client cert and the truststore the SubCA cert. I then tried to add the SubCA cert to the keystore too, but java just ignored it.

So this solves the hanshake failure mystery, but not my problem.

I created a separate question for that...sigh :-( why doesn't java send the client certificate during SSL handshake?

Hine

Answer 2

I think the trust store not containing the CA is the most likely issue. You can use the Java keytool to import the certificate for the site into the cacerts file doing something like:

keytool -keystore pathtocacerts -import -trustcacerts -v -alias aliasName -file root.crt

The default cacerts keystore password is changeit. The cacerts file is usually under jre/lib/security directory.

Coruscate

I'm pretty sure the truststore does contain the CA. When that didn't work i also imported the root CA certificate into the truststore...but that didn't seem to help. – Chum

Note that with SSL debug on, the contents of both the truststore and keystore get printed in the console. Also the client certificate and the CA cert (not the root one) suffice for this to work in a browser. – Chum

I tried checking the contents of the stores with the keytool and everything seems to be OK. I even tried setting both stores via commandline options on startup (i.e. -Djavax.net.ssl.keyStore=) to no avail... – Chum

Answer 3

You don't provide enough information, but I'm guessing your client truststore is not properly configured. The truststore contains the trusted certificates that are used to sign other certs, and must include the root certificate(s) for the server and client cert chains. The client keystore contains the client SSL certificate and private key.

Moke

Thanx for your reply Greg. Please tell me what more information I should provide. I tried to be as thorough as possible, but I know I might have missed something. As I wrote I set both the truststore and the keystore. I did put Xs in the logs instead of actual certificate names and stuff as I didn't want to post those online. The keystore contains the client certificate and the private key. The truststore contains both the certificate of the CA which issued the certificate and the root CA certificate. The stacktrace doesn't seem to be too helpful here (I posted it in a comment above). – Chum

It seems the client keystore gets loaded, but the client certificate is not being presented to the server and that's why the handshake fails. The log shows an empty client certificate chain. I have no idea why though :-( – Chum