EIdOSSLUnderlyingCryptoError Exception

Question

I am using Indy (IdHTTP, OpenSSL). I use this simple code to download a page

var
  IdHTTP: TIdHTTP;
begin
  IdHTTP:=TIdHTTP.Create;
  try
    IdHTTP.Get('https://ezfile.ch/?m=help&a=tos');
  finally
    IdHTTP.Free;
  end;
end;

It returns:

EIdOSSLUnderlyingCryptoError exception "Error connecting with SSL.
    error:14094438:SSL routines:SSL3_READ_BYTES:tlsv1 alert internal error"

The site uses TLS 1.1, AES_128_CBC_SHA1, ECDHE-ECDSA. It should be easily reproducible.

Tried with various Delphi versions, Indy 10.6.2, various OpenSSL version. Changing SSLVersion option did not help.

What could be the problem?

Arminius

The site supports TLS 1.2, 1.1 and 1.0. – Pitfall

That is possible. Result is the same. PS: you shouldn't have removed that commented code (it is necessary in older Indy versions). – Arminius

I have just tried with original OpenSSL executable to test connection. Command: "openssl.exe s_client -connect ezfile.ch:443" and it returned almost the same ":error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error" – Arminius

If the prebuilt OpenSSL app cannot connect, then the server is likely misconfigured. Contact the server admin. – Acima

It is misterious that curl which uses probably same OpenSSL 1.0.2 works correctly. – Arminius

I can confirm that curl (and Indy) work, wget and openssl s_client produce the error (on a Linux box) – Pitfall

Thanks for the test. This is funny. wget and curl works for me, openssl and Indy not. – Arminius

Answer 1

What could be the problem?

The site appears operational to me. I could even connect with TLS 1.2.

I failed when trying to connect with SSLv3, however. That's a good thing.

Its may be a bug in the library. That is, its trying to connect with SSLv3, or its doing something else wrong, like omitting the server name with SNI. Or, you are loading the wrong version of OpenSSL at runtime. That is, you compiled against OpenSSL 1.0.2, but you are loading a system down level version, like 0.9.8, at run time.

You can clear the verify error:num=20 below by fetching the CA and then passing it to s_client via -CAfile.

$ echo -e "GET /?m=help&a=tos HTTP/1.1\r\nHost: ezfile.ch\r\n\r\n" | \
    openssl s_client -connect ezfile.ch:443 -tls1_1 -servername ezfile.ch -ign_eof
CONNECTED(00000003)
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Certification Authority
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=sni42046.cloudflaressl.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA 2
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA 2
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGtjCCBlugAwIBAgIRAPpZe3LxW3syzgKsuUmSJqEwCgYIKoZIzj0EAwIwgZIx
CzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNV
BAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTgwNgYDVQQD
Ey9DT01PRE8gRUNDIERvbWFpbiBWYWxpZGF0aW9uIFNlY3VyZSBTZXJ2ZXIgQ0Eg
MjAeFw0xNTAzMjUwMDAwMDBaFw0xNTA5MzAyMzU5NTlaMGsxITAfBgNVBAsTGERv
bWFpbiBDb250cm9sIFZhbGlkYXRlZDEhMB8GA1UECxMYUG9zaXRpdmVTU0wgTXVs
dGktRG9tYWluMSMwIQYDVQQDExpzbmk0MjA0Ni5jbG91ZGZsYXJlc3NsLmNvbTBZ
MBMGByqGSM49AgEGCCqGSM49AwEHA0IABGtNNuTz7vtNt80pyh8FgGuPH78FQb1D
FsR0tBKe+ygNLGysmcTkhekQeupBYMFsfHdjUy51iabkf1a2m75iqM6jggS2MIIE
sjAfBgNVHSMEGDAWgBRACWFn8LyDcU/eEggsb9TUK3Y9ljAdBgNVHQ4EFgQUo9+D
XFOnVv7lgCtpOyO+2vbluvswDgYDVR0PAQH/BAQDAgeAMAwGA1UdEwEB/wQCMAAw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCME8GA1UdIARIMEYwOgYLKwYB
BAGyMQECAgcwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLmNv
bS9DUFMwCAYGZ4EMAQIBMFYGA1UdHwRPME0wS6BJoEeGRWh0dHA6Ly9jcmwuY29t
b2RvY2E0LmNvbS9DT01PRE9FQ0NEb21haW5WYWxpZGF0aW9uU2VjdXJlU2VydmVy
Q0EyLmNybDCBiAYIKwYBBQUHAQEEfDB6MFEGCCsGAQUFBzAChkVodHRwOi8vY3J0
LmNvbW9kb2NhNC5jb20vQ09NT0RPRUNDRG9tYWluVmFsaWRhdGlvblNlY3VyZVNl
cnZlckNBMi5jcnQwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLmNvbW9kb2NhNC5j
b20wggL9BgNVHREEggL0MIIC8IIac25pNDIwNDYuY2xvdWRmbGFyZXNzbC5jb22C
ECouYm90aXNrYWNhZmUucnOCFCouZGVlZm9yZGVzaWduLmNvLnVrgg0qLmRvcmVn
YW1hLnR2gg8qLmR1dGNocG9ybi5vcmeCCyouZXpmaWxlLmNoggwqLmZhaGFtdS5u
ZXSCDCouZmFoYW11Lm9yZ4IPKi5ncmVlbmhhYml0LnVzgg8qLmp1ZGlrYXJ0dS5j
b22CFCouanVtcGZyb21yb29mLnRvZGF5ggwqLm1vYml1cy54eXqCDioubXV0aHJv
bmUubmV0ghEqLm15YWxwaGFob3N0LmNvbYIOKi5uZXdzYmVkcy5jb22CDyoucGF1
bGRpYXouYXNpYYINKi5wYXVsZGlhei5tZYIVKi5wb2thenktY2hlbWljem5lLnBs
ghMqLnNpeHR5c2l4c291bmQuY29tggsqLnNvYXdyLm9yZ4ISKi5zdGV2aWVjcmlw
cHMuY29tghoqLnN3aW5nc2V0cHJpY2Vjb21wYXJlLmNvbYILKi50dWJ5bG8ucGyC
DmJvdGlza2FjYWZlLnJzghJkZWVmb3JkZXNpZ24uY28udWuCC2RvcmVnYW1hLnR2
gg1kdXRjaHBvcm4ub3JnggllemZpbGUuY2iCCmZhaGFtdS5uZXSCCmZhaGFtdS5v
cmeCDWdyZWVuaGFiaXQudXOCDWp1ZGlrYXJ0dS5jb22CEmp1bXBmcm9tcm9vZi50
b2RheYIKbW9iaXVzLnh5eoIMbXV0aHJvbmUubmV0gg9teWFscGhhaG9zdC5jb22C
DG5ld3NiZWRzLmNvbYINcGF1bGRpYXouYXNpYYILcGF1bGRpYXoubWWCE3Bva2F6
eS1jaGVtaWN6bmUucGyCEXNpeHR5c2l4c291bmQuY29tgglzb2F3ci5vcmeCEHN0
ZXZpZWNyaXBwcy5jb22CGHN3aW5nc2V0cHJpY2Vjb21wYXJlLmNvbYIJdHVieWxv
LnBsMAoGCCqGSM49BAMCA0kAMEYCIQCdDfGZutuJWSUn3ytumOYCbpeRjkuHN5ZN
7L0AganZGQIhANpKjy+PNLTWCOzGCcva3fEbCLzpCmu2PUatE9xBl+Ck
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=sni42046.cloudflaressl.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA 2
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4166 bytes and written 408 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES128-SHA
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.1
    Cipher    : ECDHE-ECDSA-AES128-SHA
    Session-ID: ACB2C0516C9F57EE6AA973463849A53B07CAF99A6A78EE6C12AC0CDF99CC9C50
    Session-ID-ctx: 
    Master-Key: 0DCCE2B3E57E034B271296C716CFBDC4039AE4E6697A8EF560FD7423A9090ACEC3F924D331C2B8FD0FAE5631C9D8219A
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - 03 30 b1 f4 75 9a 14 f7-d5 97 03 b3 4e 4d 5e ab   .0..u.......NM^.
    0010 - d1 15 d5 09 4a 7e 88 8b-d1 ba ed 9d 20 b5 bb f4   ....J~...... ...
    0020 - 33 c0 14 44 b3 d7 1d 78-f5 f0 f5 06 dd 57 cb 58   3..D...x.....W.X
    0030 - 51 6d 0a 18 a7 97 1b d6-36 ea bd ab a3 5a bc 1e   Qm......6....Z..
    0040 - 35 47 31 4b 19 cb c5 94-ac c5 41 f1 65 6a 76 d3   5G1K......A.ejv.
    0050 - 9e b2 45 e1 3c 5d dd 4d-49 6f 2f f2 18 1b 88 45   ..E.<].MIo/....E
    0060 - 9b 9d 50 1e 66 e2 ec c9-e5 87 a1 5a b7 80 d3 60   ..P.f......Z...`
    0070 - 6d fe 3e b6 77 0b c2 ba-f9 f9 12 49 f3 55 72 02   m.>.w......I.Ur.
    0080 - b1 da 2b 4c a6 74 50 df-11 12 c9 6b 1d 2f da a8   ..+L.tP....k./..
    0090 - 4f bc c5 9e ff f1 ff 5d-9a 28 ad e9 4d 43 09 ed   O......].(..MC..
    00a0 - bb 7d d6 1d fc 39 75 1e-e2 6e 2f f4 a6 69 7e 6c   .}...9u..n/..i~l
    00b0 - 97 cd 9c 1a 77 0d 14 c7-61 f8 87 cf 24 52 60 3e   ....w...a...$R`>

    Start Time: 1430097932
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
HTTP/1.1 200 OK
Server: cloudflare-nginx
Date: Mon, 27 Apr 2015 01:18:45 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=dc202f5845fd4246ec401ee26196a7e831430097524; expires=Tue, 26-Apr-16 01:18:44 GMT; path=/; domain=.ezfile.ch; HttpOnly
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Mon, 27 Apr 2015 01:18:45 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
X-Frame-Options: sameorigin
Access-Control-Allow-Origin: *
CF-RAY: 1dd6b1fac4350874-IAD
...

Untutored

The flag "-servername" is the key. I recieve the same result. Without it I get error. It is an option for SNI (Server Name Indication) to get correct certificate. To get back to Indy - I have been able to fix the problem by providing SNI (it is not used / implemented by default). This command helps: "SSL_set_tlsext_host_name". It seems it is pending change in Indy. – Arminius

@Arminius - a Wireshark trace would have been a red flag, but you would have needed to know what to look for. Now, you will never forget to look for the SNI extension in a trace :) – Untutored

@smooty86: the current version of Indy supports client-side SNI. – Acima

Answer 2

Here is sample code on how to fix this issue using SSL_set_tlsext_host_name

It is done by creating a custom class inheriting from TIdHTTP, and calling SSL_set_tlsext_host_name with the correct parameters by using the OnStatusInfoEx event of TIdSSLIOHandlerSocketOpenSSL

This issue started appearing on all Cloudflare SSL-enabled websites about a month ago.

program Project1;

{$APPTYPE CONSOLE}

uses
  System.Classes, IdHTTP, IdSSL, IdSSLOpenSSL, IdSSLOpenSSLHeaders, IdCTypes;

type
  TCustomIdHTTP = class(TIdHTTP)
  public
    constructor Create(AOwner: TComponent);
  private
    procedure OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL; const AWhere, Aret: TIdC_INT; const AType, AMsg: String);
  end;

{ TCustomIdHTTP }

constructor TCustomIdHTTP.Create(AOwner: TComponent);
begin
  IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
  with IOHandler as TIdSSLIOHandlerSocketOpenSSL do begin
    OnStatusInfoEx := Self.OnStatusInfoEx;
    SSLOptions.Method := sslvSSLv23;
    SSLOptions.SSLVersions := [sslvTLSv1_2, sslvTLSv1_1, sslvTLSv1];
  end;
  inherited Create(AOwner);
end;

procedure TCustomIdHTTP.OnStatusInfoEx(ASender: TObject; const AsslSocket: PSSL; const AWhere, Aret: TIdC_INT;
  const AType, AMsg: String);
begin
  SSL_set_tlsext_host_name(AsslSocket, Request.Host);
end;
//////////////////

var
  MyHTTP: TCustomIdHTTP;
begin
  MyHTTP := TCustomIdHTTP.Create(nil);
  // Your normal Indy HTTP code here
  MyHTTP.Free;
end.

Nephelometer

Yes, I came across this issue on a server using CloudFlare certificate. This solution is already mentioned in the correct answer comment ;) – Arminius

@Arminius I know, but code is better than just an API name :) – Nephelometer

FYI, the current version of Indy supports client-side SNI, it now calls SSL_set_tlsext_host_name() for you. – Acima

This workaround fixed the issue on Windows, but the problem still occurs on MacOS. I'm using XE8 firemonkey. Any suggestions? – Occult

I am experiencing this problem as well on Lazarus / Mac OS X – Cleavland

Answer 3

This code works with the 5273 revision and OpenSSL 1.0.2 / 1.0.2a:

program HttpsGetExample;

{$APPTYPE CONSOLE}

uses
  IdHTTP, IdGlobal, SysUtils, Classes;

var
  HTTP: TIdHTTP;
  ResponseBody: string;
begin
  HTTP := TIdHTTP.Create;
  try
    try
      ResponseBody := HTTP.Get('https://ezfile.ch');
      WriteLn(ResponseBody);
      WriteLn(HTTP.ResponseText);
    except
      on E: EIdHTTPProtocolException do
      begin
        WriteLn(E.Message);
        WriteLn(E.ErrorMessage);
      end;
      on E: Exception do
      begin
        WriteLn(E.Message);
      end;
    end;
  finally
    HTTP.Free;
  end;
  ReadLn;
  ReportMemoryLeaksOnShutdown := True;
end.

Pitfall

Not for me. Code is the same. I have tested it also on another computer with the same result. – Arminius

Answer 4

The above method SSL_set_tlsext_host_name worked for me except that I had to free IOHandler in an added destructor. Otherwise an exception were raised.

Tereus