SSL Handshake Failing With 'Certificate Unknown'

Question

We have an application that is currently running via HTTP protocol. We are aiming to migrate it to HTTPS. We have made the necessary changes, but then during login to the application I am getting a "peer not authenticated" error message.

I am completely new to the SSL world, and so I Google up and have captured the Wireshark trace and the communication looks as below:

1. Client sends [SYN] to server.
2. Server sends [SYN,ACK] to client.
3. Client sends [ACK] to server.
4. Client sends the message ClientHello to the server.
5. Server sends ServerHello and then its certificate with the messages “ServerHello, Certificate, ServerHelloDone
6. Alert 61, Level Fatal, Description: Certificate Unknown // Failing here.

Please share your inputs on what could be going wrong. We are stuck here and not able to proceed further.

Compatriot

Please add a screenshot of the wireshark trace so that we know where the alert is coming from (client or server) . – Horatia

It sounds like the client can't validate the server's certificate, probably because the client doesn't know, or doesn't trust, the root certificate authority used to sign the server's certificate. The root authority must be known to the client, or the client needs to disable certificate validation (which is not good for security). – Fosterfosterage

We see this issue getting resolved if I import the server security certificate onto the client. We tried this with three different clients. Does this mean anything?. Why should a certificate that belongs to the server be installed on the client?. – Compatriot

@PavanDittakavi That means it must be self-signed, which means nobody will trust it unless explicitly configured to do so via that import procedure. Best solution is to get it signed by a CA. – Welbie

Answer 1

UPDATED

This is a strange error. The Certificate Unknown should usually be accompanied by a Alert code of 46 and not 61.

If you see, SSL Alert 61 is not mentioned in the Alert Protocol (RFC 5246)

  enum {
      close_notify(0),
      unexpected_message(10),
      bad_record_mac(20),
      decryption_failed_RESERVED(21),
      record_overflow(22),
      decompression_failure(30),
      handshake_failure(40),
      no_certificate_RESERVED(41),
      bad_certificate(42),
      unsupported_certificate(43),
      certificate_revoked(44),
      certificate_expired(45),
      certificate_unknown(46),
      illegal_parameter(47),
      unknown_ca(48),
      access_denied(49),
      decode_error(50),
      decrypt_error(51),
      export_restriction_RESERVED(60),
      protocol_version(70),
      insufficient_security(71),
      internal_error(80),
      user_canceled(90),
      no_renegotiation(100),
      unsupported_extension(110),
      (255)
  } AlertDescription;

Without looking at the trace, it is difficult to investigate further.

It Looks like the Server certificate provided in the Server Hello wasn't trusted by the client.

I would recommend to test this using cURL.exe with the -v option.

Horatia

Failure to provide a client certificate isn't really an error in TLS, and it hasn't happened here: the server has only got to ServerHelloDone. If the server 'needs' a client certificate and doesn't get one it either continues or sends a handshake_failure alert. It is a TLS protocol violation for the client to send an untrusted certificate, or one of the wrong type. – Welbie

Yea, it looks like it hasn't happened here. However, failure to provide the client cert can cause the Handshake failure. This again depends and at the moment I haven't seen the network traces to be really sure what has happened. Also 61 is not something I expected. However I will edit the post to remove that to avoid confusion. – Horatia

We see this issue getting resolved if I import the server security certificate onto the client. We tried this with three different clients. Does this mean anything?. Why should a certificate that belongs to the server be installed on the client?. – Compatriot

Is that a self signed certificate? Is the CA that issued the server certificate installed on the client CA certificate store? – Horatia

So it means it's a certificate trust issue. – Horatia