How to validate / verify an X509 Certificate chain of trust in Python?
Asked Answered
E

5

18

I am working on implementing a web application that utilizes an API. During a response, the API server sends over a link to an X509 certificate (in PEM format, composed of a signing certificate and one or more intermediate certificates to a root CA certificate ) that I must download and use to do further verification.

Before using the certificate, I need to ensure that all certificates in the chain combine to create a chain of trust to a trusted root CA certificate (to detect and avoid any malicious requests). I am having a hard time doing this in python and my research into the subject is not yielding anything useful.

The certificate is easily grabbed and loaded using requests and M2Crypto

import requests
from M2Crypto import RSA, X509

mypem = requests.get('https://server.com/my_certificate.pem')   
cert = X509.load_cert_string(str(mypem.text), X509.FORMAT_PEM)

However, validating the certificate chain is a problem. It is not feasible for me to write the certificate to disk in order to use a command line utility like openssl through something like subprocess, so it must be done through python. I also do not have any open connections and so using a connection based validation solution (like is mentioned in this answer / thread: https://mcmap.net/q/235552/-validate-ssl-certificates-with-python) will not work either.

On another thread about this problem (at https://stackoverflow.com/a/4427081) abbot explains that m2crypto is incapable of doing this validation and says that he has written an extension to allow validation (using the module m2ext) but his patch never seems work, always returning false even though I know it's valid:

from m2ext import SSL
ctx = SSL.Context()
ctx.load_verify_locations(capath='/etc/ssl/certs/') # I have run c_rehash in this directory to generate a list of cert files with signature based names
if not ctx.validate_certificate(cert): # always happens
    print('Invalid certificate!') 

There's also this answer on a similar thread here https://mcmap.net/q/740926/-how-do-i-use-m2crypto-to-validate-a-x509-certificate-chain-in-a-non-ssl-setting in which John Matthews claims to have a patch written which will do it, but unfortunately the patch link is now dead -- and anyway there is a comment on that thread stating that the patch did not work with openssl 0.9.8e.

All answers relating to validating a certificates chain of trust in python seem to either link to the dead patch or go back to m2ext.

Is there a simple, straightforward way to go about validating my certificates chain of trust in Python?

Eustacia answered 8/6, 2015 at 2:3 Comment(3)
Btw pyopenssl wraps openssl so you would not need to use it from the command line. Also worth reading if you have not already python.org/dev/peps/pep-0476Kyphosis
Thanks for the link Avi, I had not seen that before. It doesn't help me in my current predicament (it seems to be about http clients in particular) however. I don't have a problem using a command line program to do the verification if necessary -- but not if I am required to write the pem file to disk upon every request. If the openssl verify command could take in a raw string instead I could use that I suppose (even though it seems like a hacky workaround for something I thought for sure would be trivial in Python).Eustacia
see also Getting certificate chain with Python 3.3 SSL moduleMaddis
J
17

While the response of Avi Das is valid for the trivial case of verifying a single trust anchor with a single leaf certificate, it places trust in the intermediate certificate. That means that in the case where the intermediate is sent, as well as the client certificate, the entire chain is trusted.

Do not do this. The code found in pyOpenSSL's tests are flawed!

I found this thread on Python's cryptography-dev mailing lists (which links back to this answer): https://mail.python.org/pipermail/cryptography-dev/2016-August/000676.html

We see this code makes no distinction between the root_cert and the intermediate. If we look at the documentation, add_cert itself adds a trusted cert (maybe add_trusted_cert would be a better name?).

It includes examples on why this is a terrible idea. I can not stress this enough: verifying your chain by trusting the intermediates is similar to not performing any check at all.


Having said that, how do you verify a certificate chain in Python? The best alternative I found is https://github.com/wbond/certvalidator, which seems to do the job.

There are also some flawed alternatives:

This is the current state for some respectable Python cryptography libraries:

Both threads seem stale at this time.

I know this does not match the question's case, but: if you are building something using certificate validation in TLS sockets, just use the modules already available in Python. Never re-invent the wheel, especially regarding cryptography. Crypto is hard; the only easy thing about it is messing it up.

Jansenism answered 14/3, 2018 at 16:15 Comment(3)
github.com/pyca/pyopenssl/pull/948 is an updated PR of 473 and has been merged. So, maybe X509StoreContext can be safely used with the chain arg?Enjoyable
An example of X509StoreContext with chain can be found below.Enjoyable
FWIW: github.com/pyca/cryptography/issues/2381 has been resolved, and a public API has been added to Cryptography for chain verification. I've included a full example in an answer below.Plummet
P
3

As of 2024, cryptography supports X.509 certificate chain verification directly. You no longer need pyOpenSSL or any other Python OpenSSL bindings to perform chain verification.

The relevant APIs are available under cryptography.x509.verification, and an example can be found in cyptography's docs:

from cryptography.x509 import Certificate, DNSName, load_pem_x509_certificates
from cryptography.x509.verification import PolicyBuilder, Store

import certifi

from datetime import datetime

with open(certifi.where(), "rb") as pems:
   store = Store(load_pem_x509_certificates(pems.read()))

builder = PolicyBuilder().store(store)
builder = builder.time(verification_time)

verifier = builder.build_server_verifier(DNSName("cryptography.io"))

# NOTE: peer and untrusted_intermediates are Certificate and
#       list[Certificate] respectively, and should be loaded from the
#       application context that needs them verified, such as a
#       TLS socket.
chain = verifier.verify(peer, untrusted_intermediates)
Plummet answered 15/2 at 18:6 Comment(1)
nit: verification_time is unset. default is datetime.now()Maddis
K
2

I looked into pyopenssl library and found this for certificate chain validation. The following example is from their tests and seem to do what you want, which is validating chain of trust to a trusted root certificate. Here are the relevant docs for X509Store and X509StoreContext

from OpenSSL.crypto import load_certificate, load_privatekey
from OpenSSL.crypto import X509Store, X509StoreContext
from six import u, b, binary_type, PY3
root_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIIC7TCCAlagAwIBAgIIPQzE4MbeufQwDQYJKoZIhvcNAQEFBQAwWDELMAkGA1UE
BhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdU
ZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwIhgPMjAwOTAzMjUxMjM2
NThaGA8yMDE3MDYxMTEyMzY1OFowWDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklM
MRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9U
ZXN0aW5nIFJvb3QgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAPmaQumL
urpE527uSEHdL1pqcDRmWzu+98Y6YHzT/J7KWEamyMCNZ6fRW1JCR782UQ8a07fy
2xXsKy4WdKaxyG8CcatwmXvpvRQ44dSANMihHELpANTdyVp6DCysED6wkQFurHlF
1dshEaJw8b/ypDhmbVIo6Ci1xvCJqivbLFnbAgMBAAGjgbswgbgwHQYDVR0OBBYE
FINVdy1eIfFJDAkk51QJEo3IfgSuMIGIBgNVHSMEgYAwfoAUg1V3LV4h8UkMCSTn
VAkSjch+BK6hXKRaMFgxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJJTDEQMA4GA1UE
BxMHQ2hpY2FnbzEQMA4GA1UEChMHVGVzdGluZzEYMBYGA1UEAxMPVGVzdGluZyBS
b290IENBggg9DMTgxt659DAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GB
AGGCDazMJGoWNBpc03u6+smc95dEead2KlZXBATOdFT1VesY3+nUOqZhEhTGlDMi
hkgaZnzoIq/Uamidegk4hirsCT/R+6vsKAAxNTcBjUeZjlykCJWy5ojShGftXIKY
w/njVbKMXrvc83qmTdGl3TAM0fxQIpqgcglFLveEBgzn
-----END CERTIFICATE-----
""")
intermediate_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICVzCCAcCgAwIBAgIRAMPzhm6//0Y/g2pmnHR2C4cwDQYJKoZIhvcNAQENBQAw
WDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAw
DgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwHhcNMTQw
ODI4MDIwNDA4WhcNMjQwODI1MDIwNDA4WjBmMRUwEwYDVQQDEwxpbnRlcm1lZGlh
dGUxDDAKBgNVBAoTA29yZzERMA8GA1UECxMIb3JnLXVuaXQxCzAJBgNVBAYTAlVT
MQswCQYDVQQIEwJDQTESMBAGA1UEBxMJU2FuIERpZWdvMIGfMA0GCSqGSIb3DQEB
AQUAA4GNADCBiQKBgQDYcEQw5lfbEQRjr5Yy4yxAHGV0b9Al+Lmu7wLHMkZ/ZMmK
FGIbljbviiD1Nz97Oh2cpB91YwOXOTN2vXHq26S+A5xe8z/QJbBsyghMur88CjdT
21H2qwMa+r5dCQwEhuGIiZ3KbzB/n4DTMYI5zy4IYPv0pjxShZn4aZTCCK2IUwID
AQABoxMwETAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBDQUAA4GBAPIWSkLX
QRMApOjjyC+tMxumT5e2pMqChHmxobQK4NMdrf2VCx+cRT6EmY8sK3/Xl/X8UBQ+
9n5zXb1ZwhW/sTWgUvmOceJ4/XVs9FkdWOOn1J0XBch9ZIiFe/s5ASIgG7fUdcUF
9mAWS6FK2ca3xIh5kIupCXOFa0dPvlw/YUFT
-----END CERTIFICATE-----
""")
untrusted_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICWDCCAcGgAwIBAgIRAPQFY9jfskSihdiNSNdt6GswDQYJKoZIhvcNAQENBQAw
ZjEVMBMGA1UEAxMMaW50ZXJtZWRpYXRlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzAeFw0xNDA4MjgwMjEwNDhaFw0yNDA4MjUwMjEwNDhaMG4xHTAbBgNV
BAMTFGludGVybWVkaWF0ZS1zZXJ2aWNlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqpJZygd+w1faLOr1
iOAmbBhx5SZWcTCZ/ZjHQTJM7GuPT624QkqsixFghRKdDROwpwnAP7gMRukLqiy4
+kRuGT5OfyGggL95i2xqA+zehjj08lSTlvGHpePJgCyTavIy5+Ljsj4DKnKyuhxm
biXTRrH83NDgixVkObTEmh/OVK0CAwEAATANBgkqhkiG9w0BAQ0FAAOBgQBa0Npw
UkzjaYEo1OUE1sTI6Mm4riTIHMak4/nswKh9hYup//WVOlr/RBSBtZ7Q/BwbjobN
3bfAtV7eSAqBsfxYXyof7G1ALANQERkq3+oyLP1iVt08W1WOUlIMPhdCF/QuCwy6
x9MJLhUCGLJPM+O2rAPWVD9wCmvq10ALsiH3yA==
-----END CERTIFICATE-----
""")

root_cert = load_certificate(FILETYPE_PEM, root_cert_pem)
intermediate_cert = load_certificate(FILETYPE_PEM, intermediate_cert_pem)
untrusted_cert = load_certificate(FILETYPE_PEM, untrusted_cert_pem)
store = X509Store()
store.add_cert(root_cert)
store.add_cert(intermediate_cert)
store_ctx = X509StoreContext(store, untrusted_cert)
print(store_ctx.verify_certificate())
Kyphosis answered 8/6, 2015 at 22:7 Comment(8)
So, I can't get this example to work -- am I missing something? the verify_cert call is always returning None (for the certificate provided in your example and my own certificate that I tested with). I had to add the FILETYPE_PEM import at the top along with your other imports from OpenSSL.crypto. I also tried remove the extraneous newlines (after the "END CERTIFICATE" lines) in your cert strings but it is still returning none. Thank you very much for your thoughts and time on this!Eustacia
Ah! I had missed the comment staring me directly in the face in that documentation saying that None is a valid response -- my apologies. I have been trying to test with my own certificates and am getting "unable to get [local] issuer certificate" errors. On the command line I am using something like this to verify successfully: openssl verify -untrusted intermediate_cert.pem -CAfile rootcert.pem tovalidate.pem (without the -untrusted switch it fails with similar errors I am seeing) -- is it correct that in your example intermediate_server_cert is the cert that I am validating?Eustacia
Correct, intermediate_server_cert is the cert that is getting validated in this example. I think that error generally means that a particular certificate is missing somewhere in the chain. That is strange though, I have tried with a different example and it did manage to resolve for me.Kyphosis
Thank you very much for your help @Avi. I am still working on this, trying to get past that error. I have pasted the certificate that I am attempting to validate here pastebin.com/z9TbDPVi if you'd like to attempt it I'd be very grateful.Eustacia
Hey didnt get a chance to look at it, but possibly you are missing a cert somewhere in the chain, either an intermediate or the root cert? One way to debug would be to try it on the terminal first with openssl verify -CAfile RootCert.pem -untrusted Intermediate.pem UserCert.pem . Once you verify it on the terminal and are sure that you have the full chain, port it to python using X509Store and X509StoreContextKyphosis
Am I missing something? You're adding intermediate_server_cert to the trusted store before verifying it. Thus any intermediate cert, valid or not, will appear to be trustworthy.Levania
You are right, this is a completely wrong answer. Putting trust in an intermediate certificate is a very, very bad idea. I have added an answer that is correct, or at least less incorrect.Jansenism
How to correctly deal with intermediate certificates is explained here: duo.com/labs/research/chain-of-fools . Uses almost exactly the same code, I've also copied it in my answer for a more persistent archive.Constrictive
C
2

Used the excellent explanation on https://duo.com/labs/research/chain-of-fools to adapt the previous solution (from avi_das), and added some comments in code. Basically we need to only add certificates to the store when they are trusted (e.g. root certificate) or verified/trusted by another (e.g. intermediate certificate). You cannot add all certificates to the store in one go, as you need to verify each certificate along the chain with the correct certificates in the store at that moment. The code comments below are less abstract.

import OpenSSL
from six import u, b, binary_type, PY3

root_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIIC7TCCAlagAwIBAgIIPQzE4MbeufQwDQYJKoZIhvcNAQEFBQAwWDELMAkGA1UE
BhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdU
ZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwIhgPMjAwOTAzMjUxMjM2
NThaGA8yMDE3MDYxMTEyMzY1OFowWDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklM
MRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9U
ZXN0aW5nIFJvb3QgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAPmaQumL
urpE527uSEHdL1pqcDRmWzu+98Y6YHzT/J7KWEamyMCNZ6fRW1JCR782UQ8a07fy
2xXsKy4WdKaxyG8CcatwmXvpvRQ44dSANMihHELpANTdyVp6DCysED6wkQFurHlF
1dshEaJw8b/ypDhmbVIo6Ci1xvCJqivbLFnbAgMBAAGjgbswgbgwHQYDVR0OBBYE
FINVdy1eIfFJDAkk51QJEo3IfgSuMIGIBgNVHSMEgYAwfoAUg1V3LV4h8UkMCSTn
VAkSjch+BK6hXKRaMFgxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJJTDEQMA4GA1UE
BxMHQ2hpY2FnbzEQMA4GA1UEChMHVGVzdGluZzEYMBYGA1UEAxMPVGVzdGluZyBS
b290IENBggg9DMTgxt659DAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GB
AGGCDazMJGoWNBpc03u6+smc95dEead2KlZXBATOdFT1VesY3+nUOqZhEhTGlDMi
hkgaZnzoIq/Uamidegk4hirsCT/R+6vsKAAxNTcBjUeZjlykCJWy5ojShGftXIKY
w/njVbKMXrvc83qmTdGl3TAM0fxQIpqgcglFLveEBgzn
-----END CERTIFICATE-----
""")
intermediate_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICVzCCAcCgAwIBAgIRAMPzhm6//0Y/g2pmnHR2C4cwDQYJKoZIhvcNAQENBQAw
WDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAw
DgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwHhcNMTQw
ODI4MDIwNDA4WhcNMjQwODI1MDIwNDA4WjBmMRUwEwYDVQQDEwxpbnRlcm1lZGlh
dGUxDDAKBgNVBAoTA29yZzERMA8GA1UECxMIb3JnLXVuaXQxCzAJBgNVBAYTAlVT
MQswCQYDVQQIEwJDQTESMBAGA1UEBxMJU2FuIERpZWdvMIGfMA0GCSqGSIb3DQEB
AQUAA4GNADCBiQKBgQDYcEQw5lfbEQRjr5Yy4yxAHGV0b9Al+Lmu7wLHMkZ/ZMmK
FGIbljbviiD1Nz97Oh2cpB91YwOXOTN2vXHq26S+A5xe8z/QJbBsyghMur88CjdT
21H2qwMa+r5dCQwEhuGIiZ3KbzB/n4DTMYI5zy4IYPv0pjxShZn4aZTCCK2IUwID
AQABoxMwETAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBDQUAA4GBAPIWSkLX
QRMApOjjyC+tMxumT5e2pMqChHmxobQK4NMdrf2VCx+cRT6EmY8sK3/Xl/X8UBQ+
9n5zXb1ZwhW/sTWgUvmOceJ4/XVs9FkdWOOn1J0XBch9ZIiFe/s5ASIgG7fUdcUF
9mAWS6FK2ca3xIh5kIupCXOFa0dPvlw/YUFT
-----END CERTIFICATE-----
""")
untrusted_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICWDCCAcGgAwIBAgIRAPQFY9jfskSihdiNSNdt6GswDQYJKoZIhvcNAQENBQAw
ZjEVMBMGA1UEAxMMaW50ZXJtZWRpYXRlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzAeFw0xNDA4MjgwMjEwNDhaFw0yNDA4MjUwMjEwNDhaMG4xHTAbBgNV
BAMTFGludGVybWVkaWF0ZS1zZXJ2aWNlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqpJZygd+w1faLOr1
iOAmbBhx5SZWcTCZ/ZjHQTJM7GuPT624QkqsixFghRKdDROwpwnAP7gMRukLqiy4
+kRuGT5OfyGggL95i2xqA+zehjj08lSTlvGHpePJgCyTavIy5+Ljsj4DKnKyuhxm
biXTRrH83NDgixVkObTEmh/OVK0CAwEAATANBgkqhkiG9w0BAQ0FAAOBgQBa0Npw
UkzjaYEo1OUE1sTI6Mm4riTIHMak4/nswKh9hYup//WVOlr/RBSBtZ7Q/BwbjobN
3bfAtV7eSAqBsfxYXyof7G1ALANQERkq3+oyLP1iVt08W1WOUlIMPhdCF/QuCwy6
x9MJLhUCGLJPM+O2rAPWVD9wCmvq10ALsiH3yA==
-----END CERTIFICATE-----
""")

# load certificates
root_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, root_cert_pem)
intermediate_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, intermediate_cert_pem)
untrusted_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, untrusted_cert_pem)

# Trust the root certificate
store = OpenSSL.crypto.X509Store()
store.add_cert(root_cert)

# only add intermediate if it can be verified by the root
store_ctx = OpenSSL.crypto.X509StoreContext(store, intermediate_cert)
print(store_ctx.verify_certificate())
store.add_cert(intermediate_cert)

# now that root and intermediate are trused, you can verify the end certificate using the store
store_ctx = OpenSSL.crypto.X509StoreContext(store, untrusted_cert)
print(store_ctx.verify_certificate())
Constrictive answered 27/9, 2021 at 5:54 Comment(1)
this one worked like a charm for me.Cupro
C
2

Awesome answer by Paul-Armand and the chain of fools article. This works for me. Note: I need to set the validation date of the store appropriately.

However I found a way to avoid adding the untrusted certificate as trusted. I checked the pyopenssl code (https://github.com/pyca/pyopenssl/blob/main/src/OpenSSL/crypto.py) and found there's a "chain" parameter on X509StoreContext, so I tried to just put the intermediate certificate in there like so:

# .... paste Paul-Armand's example here ....

store = OpenSSL.crypto.X509Store()
store.add_cert(root_cert)

# Set time to reuse certs from Paul-Armand's example
validation_date = datetime.datetime.strptime("2016-01-01", "%Y-%m-%d")
store.set_time(validation_date)

# This is the change, added param [intermediate_cert]
store_ctx = OpenSSL.crypto.X509StoreContext(store, untrusted_cert, [intermediate_cert])

print("Untrusted chain example")
try:
    store_ctx.verify_certificate()
    print("Verify - OK")
except OpenSSL.crypto.X509StoreContextError:
    print("Verify failed - bad")

It all validates properly for me and is closer to what I use on the command line with OpenSSL (-untrusted intermediate.pem). Sure wish I figured out pyopenssl in January 2019, would have saved me years of using the "certvalidate" python alternative. I revisited because I'm moving to EdDSA25519 keys and certvalidate does not support them.

Conlin answered 9/1, 2022 at 16:50 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.