Upgrading socket to SSLSocket with STARTTLS: recv failed

Question

I am trying to upgrade a socket to an SSLSocket using STARTTLS. On InspIRCd's wiki this is how its supposed to work

>> STARTTLS
<< :test2.chatspike.net 670 nickname :STARTTLS successful, go ahead with TLS handshake
(SSL Handshake)

So in my code I've written (slightly simplified)

else if (code.equals("670")) {
    SSLSocketFactory sslSocketFactory = ((SSLSocketFactory) SSLSocketFactory.getDefault());
    SSLSocket sslSocket = (SSLSocket) sslSocketFactory.createSocket(
                        socket,
                        socket.getInetAddress().getHostAddress(),
                        socket.getPort(),
                        true);
    sslSocket.startHandshake();
    bufferedReader = new BufferedReader(new InputStreamReader(sslSocket.getInputStream(), getEncoding()));
    bufferedWriter = new BufferedWriter(new OutputStreamWriter(sslSocket.getOutputStream(), getEncoding()));
    socket = sslSocket;
}

However, with and without startHandshake() I always get a recv failed (without I just get it when I try to read from the new bufferedReader

java.io.IOException: Can't connect to server
    at org.pircbotx.PircBotX.connect(PircBotX.java:472) [Exception handling]
    at org.pircbotx.PircBotX.connect(PircBotX.java:224) [Overridden constructor]
    at org.pircbotx.impl.PircBotXExample.main(PircBotXExample.java:185)
Caused by: java.net.SocketException: Software caused connection abort: recv failed
    at java.net.SocketInputStream.read(SocketInputStream.java:150)
    at java.net.SocketInputStream.read(SocketInputStream.java:121)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1707)
    at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:122)
    at sun.security.ssl.Handshaker.kickstart(Handshaker.java:909)
    at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1423)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1288)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
    at org.pircbotx.PircBotX.connect(PircBotX.java:411) [startHandshake line]
    ... 2 more

Full log (with -Djavax.net.debug=all)

1365707641940 <<<:kenny.chatspike.net 670 PircBotX :STARTTLS successful, go ahead with TLS handshake
keyStore is : 
keyStore type is : jks
keyStore provider is : 
init keystore
init keymanager of type SunX509
trustStore is: C:\Program Files\Java\jdk1.7.0_17\jre\lib\security\cacerts
trustStore type is : jks
trustStore provider is : 
init truststore

[*SNIP* A bunch of adding trusted certificates]

trigger seeding of SecureRandom
done seeding SecureRandom
Ignoring unavailable cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
Ignoring unavailable cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Ignoring unavailable cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
Ignoring unavailable cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA256
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
%% No cached client session
*** ClientHello, TLSv1
RandomCookie:  GMT: 1365707642 bytes = { 219, 198, 250, 114, 173, 93, 144, 4, 231, 53, 192, 242, 141, 137, 219, 115, 48, 203, 218, 188, 51, 30, 184, 245, 182, 150, 132, 254 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
***
[write] MD5 and SHA1 hashes:  len = 149
0000: 01 00 00 91 03 01 51 67   0B 7A DB C6 FA 72 AD 5D  ......Qg.z...r.]
0010: 90 04 E7 35 C0 F2 8D 89   DB 73 30 CB DA BC 33 1E  ...5.....s0...3.
0020: B8 F5 B6 96 84 FE 00 00   2A C0 09 C0 13 00 2F C0  ........*...../.
0030: 04 C0 0E 00 33 00 32 C0   07 C0 11 00 05 C0 02 C0  ....3.2.........
0040: 0C C0 08 C0 12 00 0A C0   03 C0 0D 00 16 00 13 00  ................
0050: 04 00 FF 01 00 00 3E 00   0A 00 34 00 32 00 17 00  ......>...4.2...
0060: 01 00 03 00 13 00 15 00   06 00 07 00 09 00 0A 00  ................
0070: 18 00 0B 00 0C 00 19 00   0D 00 0E 00 0F 00 10 00  ................
0080: 11 00 02 00 12 00 04 00   05 00 14 00 08 00 16 00  ................
0090: 0B 00 02 01 00                                     .....
main, WRITE: TLSv1 Handshake, length = 149
main, waiting for close_notify or alert: state 1
main, Exception while waiting for close java.net.SocketException: Software caused connection abort: recv failed
main, handling exception: java.net.SocketException: Software caused connection abort: recv failed
main, SEND TLSv1 ALERT:  fatal, description = unexpected_message
    at java.net.SocketInputStream.socketRead0(Native Method)
main, WRITE: TLSv1 Alert, length = 2
main, Exception sending alert: java.net.SocketException: Software caused connection abort: socket write error
main, called closeSocket()
1365707642425 *** Disconnected.

What could I be doing wrong that would cause this error? I know this feature works because other clients can do it, but I just can't figure out how to do it from Java

Any suggestions?

Schutt

Answer 1

I did some testing and got this error when receiving data on the socket while the sslSocket is handshaking. This can happen if you for example register (send NICK/LOGIN) to the socket and not the sslSocket. Try to make the first command you make to the server after connection TLSSTART and wait for the 670/691 code before you send anything else.

The following is not producing any exceptions for me and manages to handshake without any problems:

public static void main(String[] args) throws Exception {
    Socket socket = new Socket("irc.chatspike.net",6667);
    BufferedReader reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));
    BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
    String line;
    writer.write("STARTTLS\r\n");
    writer.flush(); 
    while ((line = reader.readLine()) != null) {
        if (line.contains(" 670 ")) {
            SSLContext sc = SSLContext.getInstance("TLS");
            sc.init(null, new TrustManager[]{
                    new X509TrustManager() {
                        public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                            return null;
                        }
                        public void checkClientTrusted(
                            java.security.cert.X509Certificate[] certs, String authType) {
                        }
                        public void checkServerTrusted(
                            java.security.cert.X509Certificate[] certs, String authType) {
                        }
                    }
                }, new java.security.SecureRandom());
            SSLSocketFactory sslSocketFactory = ((SSLSocketFactory) sc.getSocketFactory());
            SSLSocket sslSocket = (SSLSocket) sslSocketFactory.createSocket(
                                socket,
                                socket.getInetAddress().getHostAddress(),
                                socket.getPort(),
                                true);
            sslSocket.startHandshake();
        }
    }

Calycle

Sending STARTTLS as the very first line still gives me recv failed. Interestingly spawning a new thread to read from the cleartext bufferedReader to see if anything comes through caused a "EOFException: SSL peer shut down incorrectly" – Schutt

After lots of testing it turns out the problem was sending anything to the server in between STARTTLS and receiving the 670 line, which I think caused the server to terminate the connection. Thanks for the code, it did prove that this was possible in Java so I didn't give up – Schutt

The problem does not "lie with you making the SSLSocket as a layer on top of a functioning Socket." That's exactly what STARTTLS is supposed to do. -1 – Perrotta

@EJP This answer, while technically wrong, proved that what I was doing is possible and gave me direction. No other answers came, so I accepted and +1 this one. – Schutt

It would be wrong to leave the incorrect first sentence uncommented on to mislead future readers. What you do with your votes and upticks is your concern. – Perrotta

@EJP I meant something else with the unclear, now removed sentence - although I think the rest made what I meant obvious. – Calycle

Answer 2

The problem here is that you are sending something other than an SSL handshake after the STARTTLS command, via the plaintext socket. Once you've issued STARTTLS and it has been accepted, use of the plaintext socket must cease.

Perrotta