SSL Handshaking Using Self-Signed Certs and SSLEngine (JSSE)

Question

I have been tasked to implement a custom/standalone Java webserver that can process SSL and non-SSL messages on the same port.

I have implemented an NIO server and its working quite well for non-SSL requests. I am having a heck of a time with the SSL piece and could really use some guidance.

Here's what I have done so far.

In order to distinguish between SSL and non-SSL messages, I check the first byte of the inbound request to see if it is a SSL/TLS message. Example:

   byte a = read(buf);
   if (totalBytesRead==1 && (a>19 && a<25)){
       parseTLS(buf);
   }

In the parseTLS() method I instantiate an SSLEngine like this:

   java.security.KeyStore ks = java.security.KeyStore.getInstance("JKS");
   java.security.KeyStore ts = java.security.KeyStore.getInstance("JKS");

   ks.load(new java.io.FileInputStream(keyStoreFile), passphrase);
   ts.load(new java.io.FileInputStream(trustStoreFile), passphrase);

   KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
   kmf.init(ks, passphrase);

   TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
   tmf.init(ts);

   SSLContext sslc = SSLContext.getInstance("TLS");     
   sslc.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);


   SSLEngine serverEngine = sslc.createSSLEngine();
   serverEngine.setUseClientMode(false);
   serverEngine.setEnableSessionCreation(true);
   serverEngine.setWantClientAuth(true);

Once the SSLEngine is instantiated, I process the inbound data using the unwrap/wrap methods using code straight out of the official JSSE Samples:

   log("----");

   serverResult = serverEngine.unwrap(inNetData, inAppData);
   log("server unwrap: ", serverResult);
   runDelegatedTasks(serverResult, serverEngine);

   log("----");

   serverResult = serverEngine.wrap(outAppData, outNetData);
   log("server wrap: ", serverResult);
   runDelegatedTasks(serverResult, serverEngine);

The first part of the handshake seems to work just fine. The client sends a handshake message and the server responds with a message with 4 records:

handshake (22)
- server_hello (2) 
- certificate (11) 
- server_key_exchange (12)
- certificate_request (13) 
- server_hello_done (14)

Next, the client sends a message with three parts:

handshake (22)
 - certificate (11)
 - client_key_exchange (16)

change_cipher_spec (20)
 - client_hello (1)

handshake (22)
 *** Encrypted Message ****

The SSLEngine unwraps the client request and parses the records but the wrap method produces 0 bytes with a handshake status of OK/NEED_UNWRAP. In other words, there's nothing for me to send back to the client and the handshake comes to a screeching halt.

This is where I am stuck.

In the debugger, I can see that the SSLEngine, specifically the ServerHandshaker, doesn't find any peer certs. This is rather obvious when I look at the certificate record from the client which is 0 bytes long. But why?

I can only assume that there's something wrong with the HelloServer response but I can't seem to put my finger on it. The server seems to be sending a valid cert but the client isn't sending anything back. Is there a problem with my keystore? Or is it the truststore? Or does it have something to do with the way I'm instantiating the SSLEngine? I'm stumped.

Couple other points:

• The keystore and truststore referenced in the code snippit above were created using the following tutorial: http://www.techbrainwave.com/?p=953
• I'm using Firefox 10 and IE 9 as the client to test the server. Same results for both web clients.
• I'm using Sun/Oracle JDK 6 and Java Secure Socket Extension (JSSE) that comes bundled with it.

I look forward to any guidance you might have but please don't tell me I'm nuts or to use Netty or Grizzly or some other existing solution. Its just not an option at this time. I just want to understand what I'm doing wrong.

Thanks in Advance!

Hallowmas

Just a stupid question: Have you generated and installed client certificates in e.g. Firefox when connecting to your web-server via HTTPS? – Panoply

Yes. FF prompted me to accept/reject the certificate. I clicked on "I accept the technical risks" and I can see the cert under Tools->Options->Advanced->Encryption->View Certificates->Server. In fact, FF won't send the second handshake message without this. – Hallowmas

That is not what I was asking. I was asking for the client certificate. SSL client authentication is optional and mostly used in corporate environments, often in combination with chip-cards. If the client does not have a certificate installed (Firefox section "Your certificate") it will not send one. – Panoply

Check this StackOverflow question: #14094046 – Trapes

Answer 1

You got NEED_UNWRAP, so do an unwrap. That in turn might give you BUFFER_UNDERFLOW, which means you have to do a read and retry the unwrap.

Similarly when you get NEED_WRAP, do a wrap: that in turn might give you BUFFER_OVERFLOW, which means you have to do a write and retry the wrap.

That wrap or unwrap might in turn might tell you to do another operation: wrap or unwrap.

Just do what it tells you to do.

Lurlinelusa

Yes, I tried to unwrap after seeing the NEED_UNWRAP but there's nothing else to read. I can see the second client handshake come across the wire in its entirety and the SSLEngine parses all of it. Unfortunately, it just doesn't produce any output for me to send back to the client. The client waits for a bit and eventually hangs up. – Hallowmas

@Hallowmas There may be nothing to read but there may still be something to unwrap. Do it the way I said, let it tell you when to read rather than assuming every unwrap needs a read. Typically the server's first response in the handshake consists of three separate messages for example, so you will get 3 NEED_WRAPs in a row: conversely the client would probably read all that in a single read but get three successive NEED_UNWRAPS. Same applies throughout the handshake. – Lurlinelusa

OK, cool. Thanks to EJP and a little more debugging, I understand what's happening now. The SSLEngine is unwrapping one record at a time. So in the second client handshake, there are actually 3 records: handshake, change_cipher_spec, and another handshake. I was only calling unwrap once so the SSLEngine only parsed the first record. It needed to process all the records before generating any bytes to send back to the client. I'm finally getting back application data! – Hallowmas

@Hallowmas very good. Make sure you continue to apply that principle to every wrap and unwrap so you can handle rehandhakes both partial and complete occuring during the conversation. Most implementations I've seen fail this test, including some well-known ones. Mine passes it ;-) – Lurlinelusa

@EJP what happens when you're presented w/ a result that has both NEED_UNWRAP and BUFFER_OVERFLOW? – Izabel

@Izabel That means that you haven't done a get() from the application receive buffer into your application. – Lurlinelusa

? I don't see a get() call in any of the examples I was looking at; when do you call it? – Izabel

@Izabel You call it when you want to read some data into your application. – Lurlinelusa

Tried it both before and after the unwrap() call, and in neither case did it seem to change anything. – Izabel

tried calling get() both before and after unwrap(), but in both cases it still had the state I described to you above. I opened a more general question on SSLEngine that doesn't include that, but gives an idea of what I"m struggling with: https://mcmap.net/q/358782/-android-sslengine-example – Izabel

@Izabel Let's stick to this thread please. NEED_UNWRAP in conjunction with BUFFER_OVERFLOW means the engine wants you to call unwrap() but there was no room in the target buffer for whatever operation you attempted, whether wrap() or unwrap(). If it was wrap(), that means you have to do a flip/write/compact on the net send buffer. If it was unwrap(), you have to do flip/get/compact on the app receive buffer. – Lurlinelusa

@EJP> whichever is fine w/ me, but I'll be honest this isn't helping me see how it's supposed to work. Is there an example or tutorial you can point to? My google-fu has been thwarted thus far. – Izabel

@Izabel what part of my last comment don't you understand? – Lurlinelusa

@EJP I have no context; there's something I'm just not getting of the overall flow of SSLEngine consuming code. There are not many examples that I could find, and I am just missing the big picture, I think. At this point, though, I'm trying to get an app out the door, so I'm just doing it myself. – Izabel

@Izabel The only 'big picture' is that it's a state machine, and you have to do what it tells you to do. It knows the SSL protocol, but it doesn't expose you to that, it just exposes you to a series of requests (NEED_*) and responses (BUFFER_*FLOW). – Lurlinelusa

Ok, but lacking any good tutorials or docs, I don't understand what the states are, represent, nor what I should do to handle them. I appreciate the snippets you've given, but I don't see how to apply them in a an actual application. – Izabel

@Izabel the states are NEED_WRAP, NEED_UNWRAP, BUFFER_UNDERFLOW, and BUFFER_OVERFLOW, and they are to be handled as I have already described. There is really nothing more to it. – Lurlinelusa

I give up. I'll just stick w/ the one I rolled myself. – Izabel

@Izabel NEED_WRAP: call wrap(), from the appSendBuffer to the netSendBuffer. NEED_UNWRAP: call unwrap() from the netRecvBuffer to the appRecvBuffer. BUFFER_UNDERFLOW when wrapping: there was nothing to wrap in the appSendHuffer, which shouldn't happen; when unwrapping it means you need to call read() into the netRecvBuffer and repeat the unwrap when there is data. BUFFER_OVERFLOW when wrapping: call write() on the netSendBuffer and repeat the wrap when it is empty; when unwrapping: the application needs to consume the appRecvBuffer before repeating the unwrap, which also shouldn't happen. – Lurlinelusa

I wrote something that hides the complexity of the SSLEngine and makes it easier to use. It might be of help to some folks visiting this thread - github.com/kashifrazzaqui/sslfacade – Autumn

@kashif It's not much practical use if it doesn't support session resumption or re-handshaking frankly, or client certificates either. Nor if it requires learning yet another API, in my opinion. It's possible to wrap the whole thing in an SSLSocketChannel, and I have done so in a commercial product. Scatter/gather I can live without, I don't think it's very usefully implemented in NIO anyway. – Lurlinelusa

@EJP Do you have a working implementation of SSL Engine? – Poeticize

@PeterPenzov Indeed, and also working implementations of both SSLSocketChannel and AsyncSSLSocketChannel. Commercial licence, contact me offsite. – Lurlinelusa

@EJP do you have e-mail? – Poeticize