disabling specific signature scheme in TLS handshake using java

Question

My java 8 application is communicating with other system via rest, secured with TLS1.2. Last 2 java patches (261, 271) has broken the connection, because they have added some backward compatibility with TLS1.3. During the handshake, it started using some newer signature scheme - rsa_pss_rsae_sha256 instead of, previously workingrsa_pkcs1_sha256 (named SHA256withRSA in java8u251), which is not working because it's trying to reach my private key (during CertificateVerify handhake step), which is protected by HSM, thus it's not available to read it.

I would like to disable this new signature scheme, because the older one is still sufficient and it worked on previous java patch and it's also used in a few other connections my application.

I have found this solution - https://bugs.openjdk.java.net/browse/JDK-8227445 but when I set this setting by direct signature scheme name rsa_pss_rsae_sha256, it didn't work. Do you know what name should I pass there to disable this specific signature scheme (or all rsa_pss_* signature schemes group)?

Ascanius

Which http client are you using? Can you post an example of your client ssl configuration? – Darr

CloseableHttpClient with SSLConnectionSocketFactory (supported protocols - "TLSv1.2", supported cipher suites TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 and a DefaultHostnameVerifier – Conflux

I was able to disable this scheme (not returning rsa_pss* algorithms in ClientHello anymore), but it still tries to use it, because while consuming CertificateRequest, the server side sends "supported signature algorithms": [ecdsa_secp256r1_sha256, rsa_pss_rsae_sha256, rsa_pkcs1_sha256,... and it's trying to connect using those algorithms in the order the server has returned. – Conflux

I would try to use java.security file, which will configure TLS for whole JVM. On JDK 8 and earlier, edit the <java-home>/lib/security/java.security. All Java TLS changes are in the roadmap - java.com/en/jre-jdk-cryptoroadmap.html. You can find in the release notes/additional information what was changed. I'm not sure if you can override default cipher order in the java somehow - it needs some debugging/testing. – Flinders

I have modified java.security file but with no luck – Conflux

The RSA PSS signature scheme differs from RSA PKCS1 only in the way the message is hashed and padded. Your HSM certainly knows how to encrypt a byte array with your private key, So maybe the solution would be to use a SSLContext (or other interface) implementation that supports RSA PSS? – Court

@MarcinMikołajczyk did you ever get a solution to this problem, I too am hitting this issue now. – Aquilar

@Paul, I have reported this issue to Oracle team and they have fixed the jdk.certpath.disabledAlgorithms property in .291 patch. I have also got an update from nCipher software (hsm) to handle the RSA PSS algorithm without having to update java. – Conflux

Thanks @MarcinMikołajczyk I appreciate the reply. I'll look into the .291 patch that could get me out of a hole in relation to this issue. – Aquilar