Ruby Net::HTTP responds with OpenSSL::SSL::SSLError "certificate verify failed" after certificate renewal

Asked 1/5, 2016 at 12:29 Answered 3/5, 2017 at 20:49

Solved ruby ssl openssl ssl-certificate net-http

We recently renewed the SSL certificate of our site, and the following occurs on Mac OS El Capitan 10.11.3:

require 'net/http'

Net::HTTP.get URI('https://www.google.com')
# => "<HTML>...</HTML>"

# The site whose certificate got renewed
Net::HTTP.get URI('https://www.example.com')
# => OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed

All my searches on Google and StackOverflow come up with answers suggesting a problem with the Ruby installation, but they seem to be related to older Ruby versions and I don't think this is the case here. Here is what I've tried:

brew update
brew upgrade openssl
rvm osx-ssl-certs update all
rvm install ruby-2.3.1 --disable-binary --with-openssl-dir="$(brew --prefix openssl)" (I did not have this version before)
rvm requirements
crlrefresh rpv to purge the OSX system wide CRL cache, per Uzbekjon's suggestion.

How can I resolve this?

Notes:

The problem does not occur on a freshly installed linux Docker container that has bare Ruby 2.2.3. So maybe it's something to do with Mac OS, or SSL local caching.
This issue might have existed before the certificate renewal. I cannot know for sure. However, the renewal did cause a similar problem with a 3rd party we're using as I discuss in this question.
The certificate installation was verified by Namecheap to be correct, online checkers show everything works, and all major browsers show the certificate as valid.

Solution

With much help from BoraMa, it is now clear what was happening. COMODO added a new root called COMODO RSA Certification Authority instead of the previous COMODO Certification Authority. The new root was not registered within Mac's keychain, causing this issue.

One way we attempted to debug this was by running:

openssl s_client -connect www.mysite.com:443

Which showed a warning verify error:num=20:unable to get local issuer certificate. This warning is not an issue, as openssl s_client does not use any certificates by default. Running the following was able to prevent the warning after downloading the certificate from COMODO into comodo.pem (index here):

openssl s_client -connect www.mysite.com:443 -CAfile comodo.pem

However, this could not and did not affect Ruby OpenSSL interface. This article made things much clearer for me, and the SSL doctor script created by its author was also helpful, as it confirmed the hypothesis. The article suggested to look at OpenSSL::X509::DEFAULT_CERT_FILE, which for me was /usr/local/etc/openssl/cert.pem. That file did not exist on my machine, which meant Apple's patch for OpendSSL was using the Keychain App. For whatever reason, importing comodo.pem into my keychain and marking it as trusted based on this post did not work.

So, the solution was to create the cert.pem file manually. I went to the keychain app, and exported all System Root certificates to system_root.pem. Then: cat system_root.pem comodo.pem > cert.pem and moving that file to /usr/local/etc/openssl/ did the trick. Running Net::HTTP.get in Ruby no longer failed.

Pedigree answered 1/5, 2016 at 12:29 Comment(3)

What certificate authority is the (new) certificate from? Or, can you perhaps show the site URL so that we can study the cert itself? Perhaps the certificate authority changed upon cert renewal and you don't have it yet on your OSX (whereas it already is on a fresh linux). – Auk 1/5, 2016 at 16:55

Thanks BoraMa. It's from COMODO. – Pedigree 1/5, 2016 at 16:57

Does a testing tool like this show that it's properly configured? Sometimes the chain file is in the incorrect order and some testers don't verify this. – Misplace 2/5, 2016 at 5:33

I would try to double-check the trusted certificate store if it contains the COMODO_RSA_Certification_Authority.pem certificate. In my (Linux) setup, the site works OK but when I temporarily remove the certificate of the COMODO cert authority from the cert store, I get exactly the same error as you (while in browsers it still works as they have their own cert stores).

BTW, the same error is also recognizable using curl as it also appears to use the same trusted cert store as ruby, so you might first ensure that the site works under curl.

In linux, the cert store is located usually in /etc/ssl/certs whereas under OSX it should probably be /System/Library/OpenSSL (see this article for other options).

You should see something like the following in the cert store directory:

root@apsara:/etc/ssl/certs$ ls -l | grep COMODO_RSA_Certification_Authority.pem
lrwxrwxrwx 1 root root     73 úno 28 10:24 COMODO_RSA_Certification_Authority.pem -> /usr/share/ca-certificates/mozilla/COMODO_RSA_Certification_Authority.crt
lrwxrwxrwx 1 root root     38 úno 28 10:24 d4c339cb.0 -> COMODO_RSA_Certification_Authority.pem
lrwxrwxrwx 1 root root     38 úno 28 10:24 d6325660.0 -> COMODO_RSA_Certification_Authority.pem

The following is a snipped of some attributes of this root CA certificate:

$ openssl x509 -in COMODO_RSA_Certification_Authority.pem -noout -text
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number:
        4c:aa:f9:ca:db:63:6f:e0:1f:f7:4e:d8:5b:03:86:9d
Signature Algorithm: sha384WithRSAEncryption
    Issuer: C=GB, ST=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO RSA Certification Authority
    Validity
        Not Before: Jan 19 00:00:00 2010 GMT
        Not After : Jan 18 23:59:59 2038 GMT
    Subject: C=GB, ST=Greater Manchester, L=Salford, O=COMODO CA Limited, CN=COMODO RSA Certification Authority
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            Public-Key: (4096 bit)
            Modulus:
                00:91:e8:54:92:d2:0a:56:b1:ac:0d:24:dd:c5:cf:
                ...
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Subject Key Identifier: 
            BB:AF:7E:02:3D:FA:A6:F1:3C:84:8E:AD:EE:38:98:EC:D9:32:32:D4
        X509v3 Key Usage: critical
            Certificate Sign, CRL Sign
        X509v3 Basic Constraints: critical
            CA:TRUE
Signature Algorithm: sha384WithRSAEncryption
     ...

The certificate can be downloaded from Comodo here (index of all certs is here).

More info: while looking into it, it turns out that there are actually two distinct certification chains for certs by the Comodo CA. One, the older one, is the one with the root CA listed above. The newer validation chain uses "External CA root" certificates in the chain. This forum post explains further, with specific instructions for OSX for marking those certs as trusted.

Sajovich answered 1/5, 2016 at 17:19 Comment(14)

Thanks BoraMa. Interestingly, curl to my site works for me without an issue. The problem is only evident when using Ruby. My /System/Library/OpenSSL/certs folder is empty, and all other locations from the article do not exist on my system. I believe Mac OS places all the certificates in the Keychain app, and my Keychain app contains the "COMODO Certification Authority" certificate. – Pedigree 1/5, 2016 at 17:44

Aha, curl uses the keychain indeed. Hmm, you've probably seen this? There are similar steps to the one you tried I guess... – Auk 1/5, 2016 at 18:17

Thanks. I haven't seen that particular one, but it speaks about a full upgrade of software packages. In my case upgrade did not cause the issue - simply the renewal of a certificate, and as far as my tests go, only when I run Net::HTTP.get for that site I receive an error.. So I'm baffled. – Pedigree 1/5, 2016 at 18:46

I understand but sorry, my OSX knowledge is rather limited so am a bit shooting to perhaps at least help to narrow the issue down... Does openssl s_client work for you? openssl s_client -host DOMAIN -port 443 -CApath 'a' - you can put anything in th CApath param as documented here (which may be also worth reading for you). Again, it works for me unless I remove the cert file from the store. – Auk 1/5, 2016 at 19:0

Oh, I did, sorry. :) Also I found out this: when I run File.dirname OpenSSL::Config::DEFAULT_CONFIG_FILE under ruby console, I get /etc/lib/ssl which on my system points to the /etc/ssl/ directory - the root of the trusted certs directory. Perhaps you'll find out yours this way. – Auk 1/5, 2016 at 19:5

Thanks BoraMa. openssl s_client does work for me. Namecheap's support tried it with openssl s_client -showcerts -connect www.mysite.com:443 and they confirmed everything looks good there. I also tried openssl s_client -host www.mysite.com -port 443 -CApath 'a' and it works with or without -CApath. I do receive in the output verify error:num=20:unable to get local issuer certificate and No client certificate CA names sent, but I also receive these for https://www.google.com, so I assume it is safe to ignore. – Pedigree 1/5, 2016 at 19:8

OpenSSL::Config::DEFAULT_CONFIG_FILE points to /usr/local/etc/openssl/openssl.cnf. The /usr/local/etc/openssl directory contains a certs directory, but it is empty :) – Pedigree 1/5, 2016 at 19:10

I do receive in the output verify error:num=20:unable to get local issuer certificate - this is actually important I think - this is equivalent to the verification error you were getting in ruby. And I got rid of this by setting the correct CApath (the error 20 turned to error 0). – Auk 1/5, 2016 at 19:15

Your suggestion seems spot on. It brought me to this article, which seems to talk exactly about that. Sadly, I was not able to resolve the error 20 code. Perhaps I don't have the right certificate in my Keychain. Do you have an idea how I can tell which certificates I need to import based on the chain? – Pedigree 1/5, 2016 at 20:39

Yes, the root CA certificate should be enough (you should not need the intermediate certs). So, in your case, it should be the cert with this DN: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority. I updated the answer with some description of the cert from my local trust store. – Auk 2/5, 2016 at 4:56

Awesome! Glad to report some progress here. Downloading the root certificate to comodo.pem and doing openssl s_client -showcerts -connect www.mysite.com:443 -CAfile comodo.pem no longer shows error code 20. I then followed the instructions you sent regarding the Keychain. Things don't seem to affect Ruby yet. I tried importing the pem file, change trust settings, as well as adding the pem file inside the chain of the original root certificate. Gotta catch a flight, will muck with this when I land. Thanks for the help so far! – Pedigree 2/5, 2016 at 6:3

Great to hear, bon voyage! :-) – Auk 2/5, 2016 at 6:35

BoraMa. I updated the question with the final steps I've taken to fully resolve the issue. Thanks so much for your help, you gave me the best lead to find the resolution for this issue, and I accepted your answer. – Pedigree 3/5, 2016 at 10:12

AmitA, thanks, glad you got it fixed, great write up of the solution! – Auk 3/5, 2016 at 12:38

It sounds like the problem is with your OSX certificates cache. I guess you updated your certificates before the old one has expired?

Try purging your OSX system wide CRL cache by running this command:

crlrefresh rpv

# p - purges cache, r - refreshes them, v - run in verbose mode

This is a built-in command-line tool that updates and maintains system-wide CRL cache. Read more about it in its man page (mand crlrefresh).

Tenebrous answered 1/5, 2016 at 16:9 Comment(7)

Thanks Uzbekjon. I tried running crlrefresh rpv and sudo crlrefresh rpv but unfortunately I still get the same error. Neither of the crlrefresh had any output. I am updating the answer to include your suggestion, though. – Pedigree 1/5, 2016 at 16:17

What is your OSX version? And, what are the new certificates? There were problems with DigiSign certs on older OSX. But, then again, it should have been fixed by rvm osx-ssl-certs update all. By the way, after you have refreshed your certificates, you should "reinstall" your openssl. Or, rehash openssl's certificates. – Tenebrous 1/5, 2016 at 16:24

I'm using Mac OS El Capitan 10.11.3. I just reinstalled openssl with brew reinstall openssl. It didn't solve the issue. I'm reinstalling ruby-2.3.1 with --disable-binaries and will let you know if it works. – Pedigree 1/5, 2016 at 16:29

Don't forget to specify --with-openssl-dir="$(brew --prefix openssl)" option while installing ruby. – Tenebrous 1/5, 2016 at 16:31

Thanks Uzbekjon. I tried with these flags, started a new terminal window, and still no luck. I can see that openssl version returns 0.9.8zg, while $(brew --prefix openssl)/bin/openssl version returns 1.0.2g. Is there a way for me to test that my Ruby binary indeed uses the right one under the hood? – Pedigree 1/5, 2016 at 16:43

0.9.8zg is preinstalled OSX version and brew installs openssl keg-only. To check which version of openssl your ruby installation uses type this in in your irb: require 'openssl'; OpenSSL::OPENSSL_VERSION. – Tenebrous 1/5, 2016 at 16:46

Thanks. OpenSSL::OPENSSL_VERSION returns OpenSSL 1.0.2g 1 Mar 2016. So no luck on getting a lead there.. – Pedigree 1/5, 2016 at 16:47

I have been pulling my hair out all morning with this error. This question and the answer led me to a solution that works for me. I'm not adding new information here, but just the specifics of what I did in case it's of use to anyone else with this error on a platform similar to mine.

I'm using:

Ubuntu 16.04
ruby 2.3.0
rails 4.2.7.1
HTTParty

I'm accessing an API secured with an COMODO SSL certificate. In my code, when I tried:

HTTParty.get(secured_url).tap{|response| puts response}

I got:

SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)

I, too, used the SSL doctor script noted above. When I ran the script (substituting my actual api server address for host), I got:

$ ruby doctor.rb host:443
/home/<redacted>/.rvm/rubies/ruby-2.3.0/bin/ruby (2.3.0-p0)
OpenSSL 1.0.2g  1 Mar 2016: /usr/lib/ssl
SSL_CERT_DIR=""
SSL_CERT_FILE=""

HEAD https://host:443
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed

The server presented a certificate that could not be verified:
  subject: <redacted>
  issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Organization Validation Secure Server CA
  error code 20: unable to get local issuer certificate

In a separate terminal, I went into my certs directory:

$ cd /etc/ssl/certs

And did (using COMODO_RSA_Organization_Validation_Secure_Server_CA derived from the issuer text, above):

<redacted>:/etc/ssl/certs$ openssl x509 -in COMODO_RSA_Organization_Validation_Secure_Server_CA.pem -noout -text
Error opening Certificate COMODO_RSA_Organization_Validation_Secure_Server_CA.pem
140455648364184:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('COMODO_RSA_Organization_Validation_Secure_Server_CA.pem','r')
140455648364184:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
unable to load certificate

I went to the COMODO site where the COMODO RSA Organization Validation Secure Server CA pem is located. I copied the certificate into a new file called COMODO_RSA_Organization_Validation_Secure_Server_CA.crt on my desktop (some instructions say to use crt extension instead of pem even though you need the pem certicate content).

Then, following these instructions, I did:

<redacted>:~/Desktop$ sudo cp COMODO_RSA_Organization_Validation_Secure_Server_CA.crt /usr/share/ca-certificates/COMODO_RSA_Organization_Validation_Secure_Server_CA.crt
<redacted>:~/Desktop$ sudo dpkg-reconfigure ca-certificates

Then I did:

sudo dpkg-reconfigure ca-certificates

And then:

<redacted>:~/Desktop$ ruby doctor.rb host:443
/home/<redacted>/.rvm/rubies/ruby-2.3.0/bin/ruby (2.3.0-p0)
OpenSSL 1.0.2g  1 Mar 2016: /usr/lib/ssl
SSL_CERT_DIR=""
SSL_CERT_FILE=""

HEAD https://host:443
OK

After which my code ran fine. Thank you, thank you, thank you!

Enact answered 3/5, 2017 at 20:49 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Solution

Recommended topics

Hot tags