SSLHandshakeException when trying to access ES instance from Docker

Question

I am trying to access 6.x ES instance using High Level REST Client 6.7.2. Access to this ES instance is provided to me via hostname (https://****.azureedge.net), username & password.

My Spring Boot application is getting data from the same ES without issues when it runs from my dev environment (IDE), but throws SSLHandshakeException as soon as I try run it from Docker container (from my development machine or K8s cluster in cloud).

Container is made with base image: FROM debian:stretch-slim & OpenJDK 11.0.2 with Spring Boot necessary modules.

I made some progress debugging with -Djavax.net.debug=all. It turns out that while running in docker image just few first steps of usual SSL handshaking happen:

Produced ClientHello handshake message
WRITE: TLS13 handshake, length = 2352
Raw write
Raw read (0000: 15 03 03 00 02 02 28    ......(   )
READ: TLSv1.2 alert, length = 2
Received alert message (
  "Alert": {
    "level"      : "fatal",
    "description": "handshake_failure"
  }
)

followed by SSLHandshakeException:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:938)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1749)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1708)
at org.elasticsearch.client.SecurityClient.getSslCertificates(SecurityClient.java:508) 
....
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:271)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:316)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)

When running from my local environment handshake looks uninterrupted:

Produced ClientHello handshake message
WRITE: TLS13 handshake, length = 460
Raw write
Raw read
READ: TLSv1.2 handshake, length = 155
Consuming ServerHello
ServerHello
Negotiated protocol version: TLSv1.3
Session initialized:  Session(1560119025211|TLS_AES_256_GCM_SHA384)
WRITE: TLS13 change_cipher_spec, length = 1
Raw write
Raw read
READ: TLSv1.2 change_cipher_spec, length = 1
Consuming ChangeCipherSpec message
Raw read
READ: TLSv1.2 application_data, length = 27
...
Raw read
READ: TLSv1.2 application_data, length = 8469
Consuming server Certificate handshake message
... // here is the list of 3 certificates with "SHA256withRSA", "SHA256withRSA", "SHA1withRSA" signature algorithms
Found trusted certificate ⇢ SHA1withRSA
...

While running locally I noticed CN=Microsoft IT TLS CA 2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US, as well as CN=Baltimore CyberTrust Root, OU=CyberTrust, O=Baltimore, C=IE as issuers, maybe this is important, but I guess it is expected considering ES host address (Azure).

At the end I wanted to emphasise that I didn't need to do anything special to make this work in my macOS Java 11.0.2 development environment.

I already tried following, but that didn't change anything:

• Changing base Docker image from "slim" to non slim version
• Using OpenJDK 11.0.1 or 11.0.2
• Added cert from host to TrustStore that JVM is using in a runtime. (I checked in Docker container that there is indeed one more cert, but considering when handshake failure happens I imagine this is irrelevant)
• tried to enforce App with: "-Dcom.sun.net.ssl.enableECC=false", "-Djdk.tls.client.protocols=TLSv1.3", "-Dhttps.protocols=TLSv1.3", didn't help

Interesting: curl from Docker image with BasicAuth "talks" with same URL without issues (handshake completes) & small query returns results. I guess that curl and JVM are using different sources of trusted CAs inside the Docker, different algorithms for handshaking etc..

Thanks in advance for any help

Encrimson

Instead of slim base image, try with full image and see if that helps – Bosnia

that unfortunately didn't change anything: same Exception after same handshaking steps – Encrimson

Can you do a curl -v <url> and see if you get same handshake error inside the docker container – Bosnia

curl to ES instance & exact same URL with BasicAuth inside the Docker worked. I learned that curl inside docker uses /etc/ssl/certs and JVM /JAVA_HOME/lib/security/cacerts as source of trusted CAs (please correct me if I am wrong), so not sure if this little progress is actually useful. – Encrimson

Yes, see if this helps rabexc.org/posts/certificates-not-working-java and this too unix.stackexchange.com/questions/456475/… – Bosnia

not sure how to get that cert-file :S – Encrimson

Let us continue this discussion in chat. – Bosnia

Answer 1

TLDR: enforcing TLSv1.2 for client in the app made handshaking complete from inside the docker

After a lot of try/fail attempts I made it work. Following things didn't make any difference:

• using non "slim" debian base image insead of "slim"
• using OpenJDK 11.0.2 instead of 11.0.1
• adding host's certificate to JVM TrustedStore while building docker image so it is available when container starts.
• enforcing com.sun.net.ssl.enableECC=false
• enforcing TLSv1.3 for https.protocols and/or jdk.tls.client.protocols
• enforcing TLSv1.2 for https.protocols

What fixed handshake with host, was enforcing TLSv1.2 for client by using -Djdk.tls.client.protocols=TLSv1.2 in Dockerfile, so app runs with this flag inside container. This allowed SSL handshake to complete as it should work anyways. For some reason actual negotiation about protocol version didn't work without enforcing lower version of protocol for client. Logs from local vs docker environment doesn't show any difference but this helped in docker.

What helped me to find out was:

• setting of javax.net.debug=ssl:handshake or even more detailed javax.net.debug=all so I could see details of handshake attempts
• confirming that "at least someone" can establish outbound communication from inside the docker by using curl to send same request as app is trying, which worked because curl somehow figured out how to proceed handshaking with host.
• pure luck

Thanks everyone for support & ideas

Encrimson

For someone else looking and being new at docker like me, in your docker file change your entrypoint to the following, it fixed it for me. ENTRYPOINT [ "java", "-Djdk.tls.client.protocols=TLSv1.2" ] – Gossoon