OpenSSL 1.0.1 handshake workaround in Ubuntu? [closed]

Question

I've encountered a serious bug in OpenSSL 1.0.1 on Ubuntu 12.04:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665452

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666051 <- dated Oct 3 2012!

The gist of it is that I'm able to connect to some servers but not others. Connecting to google works:

openssl s_client -connect mail.google.com:443 -debug -state -msg -CAfile /etc/ssl/certs/ca-certificates.crt

...
Protocol  : TLSv1.1
Cipher    : ECDHE-RSA-RC4-SHA
Session-ID: 94DB1AC8531115C501434B16A5E9B735722768581778E4FEA4D9B19988551397
Session-ID-ctx:
Master-Key: 8694BF510CD7568CBAB39ECFD32D115C511529871F3030B67A4F7AEAF957D714D3E94E4CE6117F686C975EFF21FB8708
Key-Arg   : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 100800 (seconds)
TLS session ticket:
0000 - fb 52 d6 d3 3c a8 75 e1-1f 1d f6 23 ab ce 55 44   .R..<.u....#..UD
0010 - 27 bf ad c4 7a 0d 83 c8-48 59 48 4b 39 bb 3c c7   '...z...HYHK9.<.
0020 - 01 1e ad b3 13 de 65 d4-e8 ea e4 35 89 83 55 8e   ......e....5..U.
0030 - e4 d5 9f 60 58 51 33 9b-83 34 b9 35 3d 46 cb a3   ...`XQ3..4.5=F..
0040 - 35 7b 48 5d 7b 86 5c d5-a1 14 9d 8c 3e 93 eb fb   5{H]{.\.....>...
0050 - ac 78 75 72 9b d2 bc 67-f2 fa 5b 75 80 a6 31 d8   .xur...g..[u..1.
0060 - 71 15 85 7f 55 4d dc fb-b0 b5 33 db 6d 36 8c c6   q...UM....3.m6..
0070 - e8 f9 54 7a 29 69 87 2c-dd f3 c5 cf 26 55 6f 6e   ..Tz)i.,....&Uon
0080 - 45 73 7a 1d e4 b3 be b2-92 3f 0b ed c4 1c a5 24   Esz......?.....$
0090 - 3c f0 ca a5                                       <...

Start Time: 1354063165
Timeout   : 300 (sec)
Verify return code: 0 (ok)

But connecting to facebook doesn't:

openssl s_client -connect graph.facebook.com:443 -debug -state -msg -CAfile /etc/ssl/certs/ca-certificates.crt -cipher SRP-AES-256-CBC-SHA

CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0xddd2c0 [0xddd340] (64 bytes => 64 (0x40))
0000 - 16 03 01 00 3b 01 00 00-37 03 02 50 b5 5d 75 42   ....;...7..P.]uB
0010 - c2 78 55 49 b5 2e de 4f-00 a6 a8 d5 cf 10 92 44   .xUI...O.......D
0020 - 28 62 34 d6 61 5e 88 c3-68 8b 96 00 00 04 c0 20   (b4.a^..h......
0030 - 00 ff 02 01 00 00 09 00-23 00 00 00 0f 00 01 01   ........#.......
>>> TLS 1.1  [length 003b]
    01 00 00 37 03 02 50 b5 5d 75 42 c2 78 55 49 b5
    2e de 4f 00 a6 a8 d5 cf 10 92 44 28 62 34 d6 61
    5e 88 c3 68 8b 96 00 00 04 c0 20 00 ff 02 01 00
    00 09 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state
read from 0xddd2c0 [0xde28a0] (7 bytes => 7 (0x7))
0000 - 15 03 02 00 02 02 28                              ......(
SSL3 alert read:fatal:handshake failure
<<< TLS 1.1  [length 0002]
    02 28
SSL_connect:error in unknown state
140581179446944:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:724:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 64 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE

The facebook connection either hangs after the client sends its hello buffer and never receives the server hello response, or returns with an error code if I pass in a cipher it recognizes. This happens with both -tls1 and -ssl3. I've tried every parameter to openssl I can think of.

apt-cache showpkg openssl

...
Provides:
1.0.1-4ubuntu5.5 -
1.0.1-4ubuntu5.3 -
1.0.1-4ubuntu3 -

I've also tried every parameter I can think of to curl but with no success, because it uses openssl under the hood.

I'm concerned that Ubuntu can't establish secure connections (an astounding statement, I realize). After two solid days of beating my head against this problem, I'm basically praying at this point that someone knows a workaround. I'm considering a downgrade to OpenSSL 1.0.0 or using libcurl4-dev with gnutls-dev instead. Both solutions leave a rotten taste in my mouth. Thanks in advance for any help you can provide.

P.S. all of this work is so that my server can interface with external https REST APIs. I consider this a fundamental requirement in any webserver today, no excuses.

UPDATE: Here is my output without passing a cipher. It doesn't matter if I pass -CAfile or not either:

openssl s_client -connect graph.facebook.com:443 -debug -state -msg -CAfile /etc/ssl/certs/ca-certificates.crt

CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x14ed1a0 [0x1515bf0] (226 bytes => 226 (0xE2))
0000 - 16 03 01 00 dd 01 00 00-d9 03 02 50 b6 39 78 6a   ...........P.9xj
0010 - 24 95 8e dc 62 19 37 4b-ab 77 b8 66 cd 48 ba a2   $...b.7K.w.f.H..
0020 - a1 2a f8 1d f8 c9 5d fb-9d db 84 00 00 66 c0 14   .*....]......f..
0030 - c0 0a c0 22 c0 21 00 39-00 38 00 88 00 87 c0 0f   ...".!.9.8......
0040 - c0 05 00 35 00 84 c0 12-c0 08 c0 1c c0 1b 00 16   ...5............
0050 - 00 13 c0 0d c0 03 00 0a-c0 13 c0 09 c0 1f c0 1e   ................
0060 - 00 33 00 32 00 9a 00 99-00 45 00 44 c0 0e c0 04   .3.2.....E.D....
0070 - 00 2f 00 96 00 41 c0 11-c0 07 c0 0c c0 02 00 05   ./...A..........
0080 - 00 04 00 15 00 12 00 09-00 14 00 11 00 08 00 06   ................
0090 - 00 03 00 ff 02 01 00 00-49 00 0b 00 04 03 00 01   ........I.......
00a0 - 02 00 0a 00 34 00 32 00-0e 00 0d 00 19 00 0b 00   ....4.2.........
00b0 - 0c 00 18 00 09 00 0a 00-16 00 17 00 08 00 06 00   ................
00c0 - 07 00 14 00 15 00 04 00-05 00 12 00 13 00 01 00   ................
00d0 - 02 00 03 00 0f 00 10 00-11 00 23 00 00 00 0f 00   ..........#.....
00e0 - 01 01                                             ..
>>> TLS 1.1  [length 00dd]
    01 00 00 d9 03 02 50 b6 39 78 6a 24 95 8e dc 62
    19 37 4b ab 77 b8 66 cd 48 ba a2 a1 2a f8 1d f8
    c9 5d fb 9d db 84 00 00 66 c0 14 c0 0a c0 22 c0
    21 00 39 00 38 00 88 00 87 c0 0f c0 05 00 35 00
    84 c0 12 c0 08 c0 1c c0 1b 00 16 00 13 c0 0d c0
    03 00 0a c0 13 c0 09 c0 1f c0 1e 00 33 00 32 00
    9a 00 99 00 45 00 44 c0 0e c0 04 00 2f 00 96 00
    41 c0 11 c0 07 c0 0c c0 02 00 05 00 04 00 15 00
    12 00 09 00 14 00 11 00 08 00 06 00 03 00 ff 02
    01 00 00 49 00 0b 00 04 03 00 01 02 00 0a 00 34
    00 32 00 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09
    00 0a 00 16 00 17 00 08 00 06 00 07 00 14 00 15
    00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0f
    00 10 00 11 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state

Nunatak

why -cipher SRP-AES-256-CBC-SHA ? – Berey

I was trying different ciphers just to get any response. Without a cipher, it hangs after the client hello send and I never receive any bytes back. – Nunatak

Works for me: pastebin.com/MXk7NPC5 – Gaud

Answer 1

Why are you passing -cipher SRP-AES-256-CBC-SHA when connecting to graph.facebook.com? Facebook certainly doesn't support SRP: http://srp.stanford.edu/.

Does it work if you don't pass that?

Also, can you give the IP address that you're getting? With 69.171.229.17, I can reproduce that exact ClientHello (modulo the nonce and with RC4-SHA are the only cipher save the SCSV) and I get a successful handshake.

Lastly, have you tried doing over an SSH tunnel to somewhere else? Sadly, when deploying TLS features in Chrome we have repeatedly found networking hardware that breaks TLS connections. (Although I can't think of a case where -ssl3 wouldn't fix it unless the hardware was actively trying to censor connections.)

Saxon

Not a bad start to your SO career. :-) – Slier

It doesn't matter if I pass graph.facebook.com or an IP address, it still hangs, but here's what I see: ping graph.facebook.com PING api.facebook.com (69.171.234.22) 56(84) bytes of data. 64 bytes from api-read-slb-10-08-prn1.facebook.com (69.171.234.22): icmp_req=1 ttl=242 time=30.2 ms – Nunatak

Also, I agree with you that network hardware is maybe the issue, it may have to do with an MTU below 1500. BUT, if that's the case, it's a flaw in openssl because (IMHO) security should happen at the stream level, not the link level, because there's no way to know how packets are routed in the wild. SO timed me out after 5 minutes so I had to add another comment. – Nunatak

Oh and I've updated my question, if I I don't pass a cipher, the connection hangs instead of returning an error. – Nunatak

Is perchance your network blocking ICMP? If your MTU is low, there could be a failed path MTU detection situation going on. – Gaud

It wasn't at first, but we opened ICMP in the firewall to allow ping and traceroute. Also, here is another confirmation that it's probably an MTU issue, now I'm debating whether we should fix our server or if I should lower the MTU on my Ubuntu box somehow (seems like a bad idea?): ubuntuforums.org/showthread.php?t=2024008 – Nunatak

@ceejayoz, Lol, he's probably more interested fixing real world issues. – Weaner

Answer 2

Setting the MTU on my Ubuntu box from 1500 to 1496 (due to one of our firewalls being set too low) allows me to receive a response from the server without having to reboot (be sure to call ifconfig first and write down your original MTU which should be 1500):

sudo ifconfig eth0 mtu 1496

I discovered my MTU by pinging with successively larger buffers (add 28 bytes for UDP header):

Fails for 1472 + 28 = 1500:

ping -s 1472 facebook.com
PING facebook.com (66.220.158.16) 1472(1500) bytes of data.
...

Works for 1468 + 28 = 1496:

ping -s 1468 facebook.com
PING facebook.com (69.171.229.16) 1468(1496) bytes of data.
1476 bytes from www-slb-ecmp-06-prn1.facebook.com (69.171.229.16): icmp_req=1 ttl=242 time=30.0 ms
...

With 1496 I'm now able to curl to facebook.com:

curl -v https://facebook.com
* About to connect() to facebook.com port 443 (#0)
*   Trying 66.220.152.16... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*        subject: C=US; ST=California; L=Palo Alto; O=Facebook, Inc.; CN=www.facebook.com
*        start date: 2012-06-21 00:00:00 GMT
*        expire date: 2013-12-31 23:59:59 GMT
*        subjectAltName: facebook.com matched
*        issuer: O=VeriSign Trust Network; OU=VeriSign, Inc.; OU=VeriSign International Server CA - Class 3; OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign
*        SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: facebook.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: https://www.facebook.com/
< Content-Type: text/html; charset=utf-8
< X-FB-Debug: 3vAg1O5OG9hB/EWC+gk1Kl3WLJRGmlQDaEodirWb+i0=
< Date: Wed, 28 Nov 2012 19:52:25 GMT
< Connection: keep-alive
< Content-Length: 0
<
* Connection #0 to host facebook.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

I personally think that MTU should have absolutely nothing to do with what the user sees at the stream level with TCP so I hope the OpenSSL folks fix this. I also wish that someone would invent an automagic bug submitter for bugs that are profoundly widespread and time-sucking.

Nunatak

This solved the issue on Ubuntu 14.04.3 LTS running in vmware player 12.0.0 build-2985596 (openssl package version 1.0.1f-1ubuntu2.15) . MTU lowered to 1488 in my case. – Cleancut

had similar issue on a debian server not connecting to a particular host, set MTU to 1450 and the issue was resolved – Aggri