SSL handshake with no server respone with java 1.8

Question

I am using the Progress DataDirect (glorious...) JDBC driver for Microsoft SQL Server 2012 & 2014. Both servers are completely updated windows as well as SQL server wise. Both enforce encryption.

I am now trying to connect to them with my java client application using a FIPS-compliant trust store with the appropriate java.security file, jars, unlimited strength cipher suites. The trust store contains a self-signed certificate by my own CA. Used algorithm is RSA.

Using JDK 1.7.0_21 it works flawlessly but when I change to 1.8.0_74 (most recent afaik) it fails because the SQL server does not want to reply after the ClientHello. Afterwards it just disconnects with a connection reset.

Furthermore, I have tried setting these properties, standalone, in different combinations but nothing worked for JDK 1.8.

• -Dhttps.protocols=TLSv1
• -Dcom.sun.net.ssl.enableECC=false
• -Djdk.tls.client.protocols=TLSv1

JDBC connection properties are, also on/off, different combinations:

• CryptoProtocolVersion=TLSv1.2
• HostNameInCertificate=my.host.com

I tried TLS 1.2, 1.1 and 1 but nothing worked.

As it works with JDK 1.7 I inclined to think that JDK 1.8 broke this mechanism.

The debug log with JDK 1.8 contain this log entry which is totally different than JDK 1.7

1.8:

Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_128_GCM_SHA256
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

1.7

Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384

What else can I try on my part to work around this problem? Switching back to 1.7 is unfortunately not an option for me or possible.

EDIT:

This is the last part of the -Djavax.net.debug=all:

*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1440778452 bytes = { 133, 30, 185, 72, 50, 27, 253, 188, 73, 244, 59, 197, 169, 24, 142, 134, 78, 66, 122, 216, 60, 62, 232, 121, 107, 245, 32, 53 }
Session ID:  {}
Cipher Suites: [TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
Extension server_name, server_name: [type=host_name (0), value=my.sqlserver.com]
***
[write] MD5 and SHA1 hashes:  len = 130
<HEX DUMP>
main, WRITE: TLSv1.2 Handshake, length = 130
[Raw write]: length = 135
<HEX DUMP>

The calling code is this:

Connection connection = DriverManager.getConnection(url, getProps());

  private static Properties getProps()
  {
    Properties properties = new Properties();
    properties.setProperty("User", "sa");
    properties.setProperty("Password", "password");
    properties.setProperty("EncryptionMethod", "SSL");
    properties.setProperty("HostNameInCertificate", "my.sqlserver.com");
    properties.setProperty("TrustStore", "<Path to trust store");
    properties.setProperty("TrustStorePassword", "password");
    properties.setProperty("ValidateServerCertificate", "true");
    properties.setProperty("CryptoProtocolVersion", "TLSv1.2");

    return properties;
  }

Earthly

You've mentioned that after ClientHello there is error on ServerHello <-- Do You know why it's happening ? Whats causing the problem on server side ? Don't mind ciphers yet, ciphers and keystores are important later in connection process - for now You don't have anserw from server which is Your problem for now =) – Ringtail

@Ringtail no after the ClientHello nothing happens. No response, exception or reply from the server. As if it were dead. What I can see in debug is that a WRITE TLSv1.2 handshake is done but then silence. – Earthly

Check this out : docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/…, also You could edit post and add end part of ssl debug + some code that is running it - maybe that would help ;) – Ringtail

@Ringtail Thanks, but I already used that guide without success. I added more details pls see my edited post – Earthly

Hey ! I've found : support.microsoft.com/en-us/kb/3135244 - some known issue sounds familiar. Hope this would help You a little, but still check sql server logs ;) – Ringtail

@Ringtail thanks, I worked through that but no change. Changing the registry and updating had no effect. – Earthly

Answer 1

For me, upgrading to JDK8 (update 191) failed on its stricter policy for allowed (TLS) crypto algorithms. I simply enabled 3DES_EDE_CBC again by modifying my ${jdk.home}/jre/lib/security/java.security file:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
    EC keySize < 224, 3DES_EDE_CBC

to:

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, \
    EC keySize < 224

or programmatically:

java.security.Security.setProperty("jdk.tls.disabledAlgorithms", 
    "SSLv3, RC4, MD5withRSA, DH keySize < 768, EC keySize < 224");

or in Spring:

<bean class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
    <property name="targetClass" value="java.security.Security" />
    <property name="targetMethod" value="setProperty" />
    <property name="arguments">
        <list>
            <value>jdk.tls.disabledAlgorithms</value>
            <value>SSLv3, RC4, MD5withRSA, DH keySize &lt; 1024, EC keySize &lt; 224</value>
        </list>
    </property>
</bean>

See also this answer related to SSLv3

Bugs