Android SSLSocket handshake failure in Android 6 and above

Question

I've written a server based on a Java SSLServerSocket that accepts connections and communicates to android applications via a custom binary protocol:

ServerSocket serverSocket = SSLServerSocketFactory.getDefault().createServerSocket(1234);
while (true) {
    Socket socket = serverSocket.accept();
    ...
}

I run the server with the following arguments:

-Djavax.net.ssl.keyStore=keystore.jks
-Djavax.net.ssl.keyStorePassword=<PASSWORD>

And the certificate is generated using the following tutorial which builds a public and private key set: http://judebert.com/progress/archives/425-Using-SSL-in-Java,-Part-2.html:

keytool -genkeypair -keystore keystore.jks -alias keyname
keytool -export -alias keyname -file keyname.crt -keystore keystore.jks 
keytool -importcert -file keyname.crt -keystore truststore.jks

Also, I make this compatible with android by building a truststore using bouncycastle:

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <PASSWORD> -destkeystore truststore.bks -deststoretype BKS -deststorepass <PASSWORD> -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-ext-jdk15on-1.58.jar

Download the bouncycastle provider here: https://www.bouncycastle.org/latest_releases.html

And moved the resulting truststore.bks into the raw resource folder.

On Android I use the following code to build a SSLSocketFactory which allows me to import the generated bouncycastle certificate which authenticates me against the server:

KeyStore trustStore = KeyStore.getInstance("BKS");
InputStream trustStoreStream = context.getResources().openRawResource(R.raw.truststore);
trustStore.load(trustStoreStream, "<PASSWORD>".toCharArray());

TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(trustStore);

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);

Socket socket = sslContext.getSocketFactory().createSocket("ip", 1234);
... use socket

This works well for Android versions below 6. My issue is at version 6 and higher I get an exception when trying to use the socket:

 Shutting down connection Socket[address=/ip,port=1234,localPort=321321] due to exception Handshake failed
 javax.net.ssl.SSLHandshakeException: Handshake failed
    at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:429)
    at com.example.Client.connect(Client.java:97)
    at com.example.Client.start(Client.java:60)
    at com.example.BackendServiceFactory$2.call(BackendServiceFactory.java:136)
    at com.example.BackendServiceFactory$2.call(BackendServiceFactory.java:130)
    ...
 Caused by: javax.net.ssl.SSLProtocolException: SSL handshake terminated: ssl=0xe69ec900: Failure in SSL library, usually a protocol error
 error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE (external/boringssl/src/ssl/s3_pkt.c:641 0xe2d10880:0x00000001)
 error:1000009a:SSL routines:OPENSSL_internal:HANDSHAKE_FAILURE_ON_CLIENT_HELLO (external/boringssl/src/ssl/s3_clnt.c:800 0xe6ea5af3:0x00000000)
    at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
    at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:357)
    ... 24 more

I'm not sure whats going on here. There seems to be a misstep along the way dealing with client certificates, could this be a mismatch of cipher suites?

I've put together a minimal example with a Java server, Java client and Android client to help diagnose this issue here:

https://github.com/johncarl81/androidCA

Bartholemy

are you using open SSL ? – Mucoviscidosis

Have you tried setting the subject alternative name on the certificates as shown at #8745107 – Michaelemichaelina

@chandrakantsharma: No – Bartholemy

@sapensadler: I can try that, but I'm not sure how that would make a difference. – Bartholemy

The only reason I suggest it, is because chrome 58 no longer uses the Common Name but uses the SAN to identify certificates. I'm guessing that as they're both google, android may have followed suit. support.google.com/chrome/a/answer/7391219?hl=en I could be completely wrong, but it might be worth a shot. It could also explain why it works on previous versions, but not after 6 if the behavior has changed. – Michaelemichaelina

Answer 1

I figured this was going to be a simple fix. Seems that I needed to specify the key algorithm in the very first keytool command:

keytool -genkeypair -keystore keystore.jks -alias keyname -keyalg RSA

This generates a 2048 bit RSA key which is compatible with versions of android < 6 and >= 6.

Bartholemy