How to force older debian to forget about DST Root CA X3 Expiration and use ISRG Root X1 - SSL certificate problem: certificate has expired

Asked 1/10, 2021 at 15:56 Answered 1/10, 2021 at 19:29

Solved debian certificate lets-encrypt root-certificate

This relates to DST Root CA X3 Expiration (September 2021)
When searching online for a fix to apply on an older server (Debian 8 in my case) that does call to sites encrypted with letsencrypt with curl, they now seem to fail with the following message:

Example:

curl -fsSL https://deb.nodesource.com/setup_14.x | bash -

Fails silently, then trying it manually and removing the silent flag and bash pipe like this:

curl -L https://deb.nodesource.com/setup_14.x

curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Trying the following commands doesn't solve the issue:

apt update
apt install -y ca-certificates openssl
update-ca-certificates

What can I do!? (answering my own question) ⬇️

Knucklebone answered 1/10, 2021 at 15:56 Comment(0)

disclaimer; I'm no security expert (I know things, but you do you). Make sure you understand what you do before applying whatever fix shared here

Fix by upgrading your instances

Upgrade your instances. This problem won't happen on Debian 9 or higher.

In the following example, I had this problem on a ruby:2.4.1 docker image which is based on Debian 8 (could be considered old). Upgrading to more recent docker image fixes this issue. Uprading to a more recent Debian version should also fix the issue.

I confirmed it does not happen when using ruby:2.7.0 docker image based on Debian 11 as shown here:

docker run --rm -it ruby:2.7.4 bash -c "cat /etc/issue"
Debian GNU/Linux 11 \n \l

Fix for Debian 8 by commenting DST_Root_CA_X3.crt from /etc/ca-certificates.conf

Even if ISRG Root X1 is in place, if DST Root CA X3 is still present and in use, its verification seems to happen first so we can get rid of it by doing this:

install ca-certificates package
comment /mozilla/DST_Root_CA_X3.crt line from /etc/ca-certificates.conf
make sure /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt is there (it should be)
update ca-certificates with update-ca-certificates

Same as above, but using scripts directly on your instance

cat /etc/issue
Debian GNU/Linux 8 \n \l

sudo apt install -y ca-certificates
sudo sed -i '/^mozilla\/DST_Root_CA_X3.crt$/ s/^/!/' /etc/ca-certificates.conf
sudo update-ca-certificates

Example fix in a Dockerfile:

FROM ruby:2.4.1 # uses debian 8

RUN apt update -qq \
    && apt install -y ca-certificates \
    && sed -i '/^mozilla\/DST_Root_CA_X3.crt$/ s/^/!/' /etc/ca-certificates.conf \
    && update-ca-certificates \
    && rm -rf /var/lib/apt/lists/*

Fix interactively using dpkg-reconfigure ca-certificates

As stated in the comments, you can also fix this interactively using the following command on the instance (requires ca-certificates package installed):

dpkg-reconfigure ca-certificates

Then disable mozilla/DST_Root_CA_X3.crt from the list.

Conclusion

If you'd like to learn more, you should read Scott Helme's post: Let's Encrypt's Root Certificate is expiring!

You can now curl letsencrypt sites safely with these safety glasses: 🥽

Knucklebone answered 1/10, 2021 at 15:56 Comment(6)

Thanks, this was really useful and helped me fix the problems I was hitting. However you don't need to delete any files, you only need to update the /etc/ca-certificates.conf to prefix the cert with a ! and update the certificates (ie. sed -i '/^mozilla\/DST_Root_CA_X3.crt$/ s/^/!/' /etc/ca-certificates.conf && update-ca-certificates). Alternatively run dpkg-reconfigure ca-certificates and use the wizard to disable the cert. – Quartersaw 2/10, 2021 at 6:56

Thanks for your comment, I've updated the answer to match what you just shared, definitely better :) – Knucklebone 3/10, 2021 at 15:58

On alpine, I tried your solution of commenting the old DST X3 CA cert /etc/ca-certificates.conf and verified that /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt is there. But I get the error "unable to get the issuer certiticate from axios that's making a call to HTTPS endpoint on the nodejs app running on the container. – Suspense 5/10, 2021 at 15:55

@SagarKalburgi which version of alpine are you using? I tried the latest and it worked fine without the fix. example: docker run --rm -it alpine:latest sh -c "apk add curl && curl https://deb.nodesource.com/setup_15.x" works, maybe upgrading to a more recent alpine image would be a better fix – Knucklebone 6/10, 2021 at 15:44

Thank you, removing the old mozilla/DST_Root_CA_X3.crt via dpkg-reconfigure ca-certificates also fixed an issue with upgrading Jenkins on a an Ubuntu 14 machine. – Lightly 13/1, 2022 at 10:29

finally the right answer ! – Earthly 14/2, 2023 at 5:29

It doesn't get clear how "old" your Debian is.

Having the latest Linux OS will not cause this issue. Consider before everything upgrading your distro to the latest version!

The following worked on my older Debian and Ubuntu systems.

To clarify the issue is with CA Root X3 certificate part of the "cacert-bundle". As of today the "cacert-bundle" can be found here: https://curl.se/docs/caextract.html as part of the bundle https://curl.se/ca/cacert.pem.

The expired certificate is:

Certificate:
    Data:
    Version: 3 (0x2)
    Serial Number:
    44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
    Signature Algorithm: sha1WithRSAEncryption
    Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
    Not Before: Sep 30 21:12:19 2000 GMT
    Not After : Sep 30 14:01:15 2021 GMT
    Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
    Subject Public Key Info:
    Public Key Algorithm: rsaEncryption
    Public-Key: (2048 bit)

Which is used to verify peer in curl calls to websites using Let's Encrypt issued certificates.

Find which CA Certificates bundle is being used by CURL:

strace curl https://www.google.com |& grep open

Result:

root@debian:/tmp# strace curl https://www.google.com |& grep open
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/x86_64-linux-gnu/libcurl.so.4", O_RDONLY|O_CLOEXEC) = 3
...
...
open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3
open("/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 4

Near the end you should see the certificate bundle. In this case: /etc/ssl/certs/ca-certificates.crt

An option to fix this issue is to replace that CA certificate bundle, but that may not be a global solution for your OS and other affected apps and packages (E.g.: php libcurl).

Now to dissect the Root CA bundle certificate, use:

openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-certificates.crt | openssl pkcs7 -print_certs -text -noout | grep "Not After"

NOTE: The cert location (/etc/ssl/certs/ca-certificates.crt) in the command is from the "strace" step above.

This will return long list with expiration dates for all Root CA certificates in the bundle.

root@L36630:/tmp# openssl crl2pkcs7 -nocrl -certfile cacert.pem | openssl pkcs7 -print_certs -text -noout | grep "Not After"
        Not After : Sep 30 04:20:49 2023 GMT
        Not After : Sep 30 14:01:15 2021 GMT
        Not After : Sep 22 11:22:02 2030 GMT
        Not After : Sep 15 16:00:00 2025 GMT
        ...
        ...

In this case, our expired certificate is "Not After : Sep 30 14:01:15 2021 GMT".

You can remove the | grep "Not After" to see the full decrypted list.

This the step-by-step method to troubleshoot the cacert bundle. If this happens in the future you know what to do.

Now let's get to fixing the issue.

FIX

Let’s Encrypt originally used the “DST Root CA X3” CA Root certificate. Let's encrypt now uses “ISRG Root X1” and “ISRG Root X2” as Root CA’s and “Let’s Encrypt R3” as an intermediate certificate.

To fix this issue, you need to add the 2 new Root CAs to your server or device:

ISRG Root X1 (Or ISRG Root X1 DER Format)
ISRG Root X2 (Or ISRG Root X2 DER Format)

Intermediate Certificate (PEM format):

Let’s Encrypt R3 (Or Let’s Encrypt R3 DER Format)

Install Root CA Certificate on Linux:

sudo mkdir /usr/local/share/ca-certificates/extra

sudo cp root.cert.pem /usr/local/share/ca-certificates/extra/root.cert.crt

sudo update-ca-certificates

NODEJS

Node.js 7.3.0 (and the LTS versions 6.10.0 and 4.8.0) added NODE_EXTRA_CA_CERTS environment variable for you to pass the CA certificate file.

$ export NODE_EXTRA_CA_CERTS=[custom Root CA certificate file path]

Alternatively you can blacklist/remove the DST certificate from the CA cert bundle for your OS.

DEBIAN/UBUNTU

You can use the dpkg to reconfigure in Ubuntu/Debian.

sudo dpkg-reconfigure ca-certificates

You will see terminal window with "ca-certificates configuration" (ca-certificates configuration image 1). Select "yes" to trust new certificates from certificate authorities.
Press "OK"
The window will close and in the next step you will see the actual list with CA certs (ca-certificates configuration image 1). Find the cert by name "DST_Root_CA_X3.crt", uncheck it and press "OK".
Now you will see "Updating certificates in /etc/ssl/certs...". Wait until the process finishes.

Next step retest your curl call from the terminal. You should no longer see the CURL error:

root@debian:/tmp# curl https://example.com
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Another option (Ubuntu/Debian):

The list of CAs is stored in the file /etc/ca-certificates.conf. You can edit this file manually and run:

sudo update-ca-certificates

To automate it:

# Make sure the ca-certificates.conf location is correct 
sed '/DST_Root_CA_X3.crt/d' /etc/ca-certificates.conf > /tmp/cacerts.conf && mv /tmp/cacerts.conf /etc/ca-certificates.conf
update-ca-certificates

RHEL/CENTOS

In RedHat add that cert to the ca-trust blacklist: /etc/pki/ca-trust/source/blacklist

Create the dstrootca.pem file:

DST Root CA X3
==============
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/MSQwIgYDVQQK
ExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMTDkRTVCBSb290IENBIFgzMB4X
DTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVowPzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1
cmUgVHJ1c3QgQ28uMRcwFQYDVQQDEw5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmT
rE4Orz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEqOLl5CjH9
UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9bxiqKqy69cK3FCxolkHRy
xXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40d
utolucbY38EVAjqr2m7xPi71XAicPNaDaeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0T
AQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQ
MA0GCSqGSIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69ikug
dB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXrAvHRAosZy5Q6XkjE
GB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZzR8srzJmwN0jP41ZL9c8PDHIyh8bw
RLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubS
fZGL+T0yjWW06XyxV3bqxbYoOb8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

Add the file to your blacklist location: /etc/pki/ca-trust/source/blacklist
Run sudo update-ca-trust

Source: https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

Let's Encrypt formal article: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Martyrology answered 1/10, 2021 at 19:29 Comment(7)

I really like the trick with strace curl to figure out which configuration files are being used. Thanks for sharing that and linking other sources! :D 🙌 – Knucklebone 1/10, 2021 at 20:7

You're welcome! Hope it helps! Just added how to decrypt the ca-bundle and see expiration dates. – Martyrology 1/10, 2021 at 21:2

I tried adding the new root CAs & the intermediate CA you mentioned, but somehow the axios on my nodejs application keeps throwing the error "unable to get issuer certificate" – Suspense 5/10, 2021 at 3:2

@SagarKalburgi here's a solution for your problem: #69414979 – Martyrology 5/10, 2021 at 16:50

thanks, but somehow that didn't work out either :( – Suspense 5/10, 2021 at 18:50

Sorry to hear that! I saw that some folks on that thread reported it working. Maybe ask them how they did it? – Martyrology 5/10, 2021 at 19:35

Turns out you cannot just remove the line from the /etc/ca-certificates.conf, update-ca-certificates doesn't know it was there in the first place. You have to prefix it with ! to tell update-ca-certificates to actually remove it, see answer from @GabLeRoux. – Pushbike 2/2, 2022 at 16:13

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Fix by upgrading your instances

Fix for Debian 8 by commenting DST_Root_CA_X3.crt from /etc/ca-certificates.conf

Fix interactively using dpkg-reconfigure ca-certificates

Conclusion

Recommended topics

Hot tags