SSL Handshaking With Older Clients Using SSLEngine (JSSE)

Question

This is a follow-up question to "SSL Handshaking Using Self-Signed Certs and SSLEngine (JSSE)".

I have implemented a NIO Webserver that can process SSL and non-SSL messages on the same port. In order to distinguish between SSL and non-SSL messages, I check the first byte of the inbound request to see if it is a SSL/TLS message. Example:

byte a = read(buf);
if (totalBytesRead==1 && (a>19 && a<25)){
    parseTLS(buf);
}

In the parseTLS() method I instantiate an SSLEngine, initiate the handshake, wrap/unwrap messages, etc. Everything seems to work fine for most modern web browsers (Firefox 10, IE 9, Safari 5, etc).

Problem is that older web browsers like IE 6 and libraries like Java's URLConnection class seem to initiate the SSL/TLS handshake differently. For example, the first few bytes from IE 6 look something like this (hex values):

80 4F 01 03 00 ...

If I pass the message to the SSLEngine, it doesn't seem to recognize the message and throws an Exception.

javax.net.ssl.SSLException: Unsupported record version Unknown-0.0

So what exactly is IE 6 and Java's URLConnection class sending over? Is this a valid SSL/TLS message that the JSSE SSLEngine can support? Do I have to do some pre-processing or negotiate with the client to send a different message?

Thanks in advance!

UPDATE

Thanks to Bruno and EJP and some further debugging I have a much better understanding of what's going on. As Bruno correctly pointed out, the IE6 and Java 6 clients are sending over a SSLv2 ClientHello. Contrary to one of my earlier comments, the SSLEngine in Java 1.6 can in fact unwrap the SSLv2 message and generate a valid response to send back to the client. The SSLException I reported earlier was an error on my side and has nothing to do with the SSLEngine (I incorrectly assumed that the client was done sending data over and I ended up with an empty ByteBuffer when the SSLEngine was expecting more data to unwrap).

Quechuan

Answer 1

This looks like an SSLv2 Client Hello (see TLS specification):

TLS 1.1 clients that support SSL Version 2.0 servers MUST send SSL Version 2.0 client hello messages [SSL2]. TLS servers SHOULD accept either client hello format if they wish to support SSL 2.0 clients on the same connection port. The only deviations from the Version 2.0 specification are the ability to specify a version with a value of three and the support for more ciphering types in the CipherSpec.

• 80 4F is the length and the high bit must be set to 1 (see msg_length description).
• 01 is the message type (Client Hello)
• 03 00 is the highest supported version (SSLv3 here)

Since Java 7, this is now disabled by default.

EDIT:

Just to clarify, this isn't really an SSLv2 Client Hello, this is a Client Hello for SSLv3 in the SSLv2 format. In this case, the server will reply with a (proper) SSLv3 Server Hello (corresponding to the 03 00 requested version number). The same also works for TLS 1.0, 1.1 and 1.2, although the usage of this format is progressively deprecated.

A JSSE 7 SSLServerSocket will still understand such a Client Hello and reply appropriately with the SSLv3/TLS1.x Server Hello.

Otherworld

Thanks. And according to the [docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/… reference guide) SUN's SSLEngine doesn't support SSLv2. So what should I send back to the client? Just shut down the connection? – Quechuan

It doesn't, but it does support higher versions of SSL/TLS initiated with this SSLv2 ClientHello (provided the supported version it indicates is SSLv3 or TLS 1.x): it would reply with an SSLv3 ServerHello to that message starting with 80 4F 01 03 00. – Otherworld

Sorry I hit the enter key too early. So you're saying I should respond to the SSLv2 ClientHello with a SSLv3 ServerHello? – Quechuan

Yes, depending on the version supported (here 03 00 -> SSLv3, 03 01 -> TLSv1.0, 03 02 -> TLSv1.1, ...). – Otherworld

What api do you suggest I use to generate the response? The unwrap method in the SSLEngine didn't seem to work for me. – Quechuan

I'm not sure, unwrap works for me with JSSE 7: the following wrap produces a TLSv1 ServerHello as expected. Perhaps try to give it the entire ClientHello message in one pass. In your example, that would be 0x4F + 2 bytes (this will vary depending on the value of msg_length of course). – Otherworld

@Quechuan As per our other conversations, you only have to call wrap() and unwrap() as dictated by the engine, and do reads and writes when it tells you. It understands the SSLv2Hello message: all you have to do is recognize it as beginning an SSL conversation. – Laszlo