WWW::Mechanize SSL connect attempt failed for https get

L

2

3

I am trying to retrieve a Web page using WWW::Mechanize and it is failing with SSL connect error. How do I fix the issue? I am running ActivePerl 5.20.2 on Windows 10 x64.

Here is the script I executed:

perl -MIO::Socket::SSL=debug4 -MWWW::Mechanize -e 'WWW::Mechanize->new()->get("https://fundresearch.fidelity.com/mutual-funds/fees-and-prices/316343201")'

The output is as follows:

DEBUG: .../IO/Socket/SSL.pm:2649: new ctx 98842176
DEBUG: .../IO/Socket/SSL.pm:562: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:564: socket connected
DEBUG: .../IO/Socket/SSL.pm:586: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:619: using SNI with hostname fundresearch.fidelity.com
DEBUG: .../IO/Socket/SSL.pm:654: request OCSP stapling
DEBUG: .../IO/Socket/SSL.pm:673: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:686: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:689: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:699: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:709: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:729: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:686: call Net::SSLeay::connect
DEBUG: .../IO/Socket/SSL.pm:2552: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2505: ok=0 cert=102327360
DEBUG: .../IO/Socket/SSL.pm:689: done Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:692: SSL connect attempt failed

DEBUG: .../IO/Socket/SSL.pm:692: local error: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
DEBUG: .../IO/Socket/SSL.pm:695: fatal SSL error: SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
DEBUG: .../lib/Net/HTTPS.pm:69: ignoring less severe local error 'IO::Socket::IP configuration failed', keep 'SSL connect attempt failed error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed'
DEBUG: .../IO/Socket/SSL.pm:2682: free ctx 98842176 open=98842176
DEBUG: .../IO/Socket/SSL.pm:2687: free ctx 98842176 callback
DEBUG: .../IO/Socket/SSL.pm:2694: OK free ctx 98842176
Error GETing https://fundresearch.fidelity.com/mutual-funds/fees-and-prices/316343201: Can't connect to fundresearch.fidelity.com:443 at -e line 1.

I am able to successfully retrieve the Web page using curl.

Lemar answered 6/1, 2018 at 18:37 Comment(3)

Can you get any other https URLs? What versions of WWW::Mechanize and IO::Socket::SSL are you using? – Lei 6/1, 2018 at 18:40

You could google the actual error message. (Why do you enable debugging when you ignore the debug output?) See #17085386 for example. – Accentuation 6/1, 2018 at 18:44

Schwern, I am able to access other https URLs. For example, access to google.com succeeds. – Lemar 6/1, 2018 at 18:55

L

4

Works For Me™ with IO::Socket::SSL 2.052, WWW::Mechanize 1.86, and Net::SSLeay 1.80. I suspect you need to upgrade Net::SSLeay. I'd suggest upgrading all of them.

The differences start here. Yours considers the cert to not be ok.

DEBUG: .../IO/Socket/SSL.pm:2552: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2505: ok=0 cert=102327360

But mine does. The more verbose output is because of my upgraded Net::SSLeay.

DEBUG: .../IO/Socket/SSL.pm:2722: did not get stapled OCSP response
DEBUG: .../IO/Socket/SSL.pm:2675: ok=1 [2] /C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2

That process is handled by Net::SSLeay. It's possible your version of Net::SSLeay is incompatible with your OpenSSL C library. There have been a lot of fixes for compatibility with OpenSSL 1.1 since ActivePerl 5.20.2 came out.

Lei answered 6/1, 2018 at 18:59 Comment(4)

"It's indicative of a bug that it will return a memory address." - No, it's indicative of an older version of IO::Socket::SSL. Before 2.018 (released 08/2015) it just showed the memory address here. See github.com/noxxi/p5-io-socket-ssl/commit/… – Alibi 6/1, 2018 at 19:12

@SteffenUllrich Thanks, I've removed that speculation. – Lei 6/1, 2018 at 19:17

Thanks Schwern. The issue was indeed with the library versions. I upgraded the 3 packages to the latest versions available from ActivePerl and that resolved the issue. – Lemar 6/1, 2018 at 19:40

@scorpio: thanks to Schwern for the quick answer which got the problem fixed for you. If you are interested in the details of the underlying problem and what other options are for fixes you might take a look at my analysis. – Alibi 6/1, 2018 at 20:3

A

6

TL;DR: upgrade Perl or at least the Mozilla::CA module or use SSL_ca_file with your own trust store.

I suspect the problem is because you are using an old version of Perl and specifically the old version of the Mozilla::CA module included with this Perl. Looking at the certificate chain for this site (for example at SSLLabs) you'll find that it looks the following:

[0] CN=fundresearch.fidelity.com
[1] CN=Entrust Certification Authority - L1K
[2] CN=Entrust Root Certification Authority - G2   -- selfsigned

The last certificate in this trust chain is self-signed, i.e. the trust in this certificate comes from the certificate itself. This is an obvious misconfiguration of the the server and that's why this certificate gets ignored when validating the trust chain.

The issuer CA for CN=Entrust Certification Authority - L1K is CN=Entrust Root Certification Authority - G2, i.e. exactly the certificate which got sent by the server but which got ignored because one should not base the trust only on what the server sends. This means there need to be an instance of this root CA in the local trust store.

Modern browsers and OS have this CA in the trust store and that's why you can access this site with a browser. But, WWW::Mechanize (based on LWP::UserAgent) does not use the systems trust store (at least on Windows). Instead the trust store is provided by the Mozilla::CA module which from time to time takes a copy of the trust store from Mozilla (i.e. Firefox).

Based on the version of your Perl distribution you probably use version 20141217 of Mozilla::CA. This version does not yet include the CA Entrust Root Certification Authority - G2 as trusted. This CA is only included with the next version 20150826. And since the CA is not considered trusted the validation will fail.

There are several fixes possible.

One would be to upgrade the version of Perl. This is maybe the best option since this also includes fixes to OpenSSL when dealing with multiple possible trust path - which is a problem you might run into at other sites (see here for more details).
Another way would be to just upgrade the Mozilla::CA module. But this might actually cause problems with some sites if you still use the old Perl where the OpenSSL issue described above is not fixed.
Or you could add the missing certificate to your cacert.pem (the one provided by Mozilla::CA).
And finally you could use the ssl_opts with SSL_ca_file to set your own trust store which includes the missing CA certificate.

Alibi answered 6/1, 2018 at 19:57 Comment(0)

L

4