Can't receive peer certificate in Python client using OpenSSL's ssl.SSLContext()

Question

I am a Windows user. I use Python 3.6.5 and I import this version of OpenSSL OpenSSL 1.0.2k.

I need to write a script for a python TLS client that I can customize in terms of the supported TLS versions and ciphersuites and other configurations. The client should be able to make connections with self-signed certificates. Therefore, I believe I should use: ssl.SSLContext() to create my context and not ssl.create_default_context().

However, with the following script, I can never get the peer's certificate. Please, provide clear answers with code as otherwise I tried many solutions and looked at previous posts with no hope.

context = ssl.SSLContext() # ssl.create_default_context() 
#context.verify_mode = ssl.CERT_NONE
#context.check_hostname = True
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
domain="google.com"
ssl_sock = context.wrap_socket(s, server_hostname=domain)
ssl_sock.connect((domain, 443))

print("====== peer's certificate ======")
try:
    cert = ssl_sock.getpeercert()
    print ("issued to:", dict(itertools.chain(*cert["subject"]))["commonName"])
    print ("issued by:", dict(itertools.chain(*cert["issuer"]))["commonName"])
    print("issuance date:", cert["notBefore"])
    print("expairy date: ", cert["notAfter"])
    if (cert == None):
        print("no certificate")

except Exception as e:
    print("Error:",e)
ssl_sock.close()

The problem is that I do not receive the peer's certificate when I use ssl.SSLContext() but when I use ssl.create_default_context() it is received correctly. However, I need to be able to receive self-signed certificates (i.e. unverified certificates) that's why I have to use ssl.SSLContext().

Thanks for the solution posted. But I need to parse the certificate even if it is not verified (self-signed). I trust this certificate and I need its info. I looked at several posts including this one. I did these steps:

1. I took the .pem content of my server's certificate.
2. I navigated to: C:\Python36\Lib\site-packages\certifi
3. I opened cacert.pem which is placed in the directory (step 2)
4. I added my server's cert .pem content which starts with: -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----

I get this error:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

Listerism

Answer 1

Here is a 9 line snippet that grabs cert data from any url.

import ssl
import socket

def getcertmeta(url="",port=443):
    hostname = socket.gethostbyaddr(url)[0]
    context = ssl.create_default_context()
    with socket.create_connection((hostname, port)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            return ssock.context.get_ca_certs()

Tin Foil Note: Something fishy is going with this cert stuff. They are the source of all kinds of problems but tech companies are obsessed with them. They provide no security benefit, every single cert authority has been hacked(except LetsEncrypt iirc) this is irrefutable and can be looked up and found online in idk 5mins, there's something else going on but they are so damn complicated and needlessly complex that its very difficult to figure out what. Imo they are being used along with intel ME to backdoor server data to central locations(like the hack revealed by wileaks v7), the certs on my phone included like 3-4 governments, just straight up says Japanese Government or Chinese Government and it has a bunch of intel agency companies like 'starlight technologies' its all very shady imo.

EDIT2: The above snippet is wrong, it only gets the certificate authority certs not the actual url, here is a 13 line snippet to get the cert for the actual url.

import socket,ssl
from contextlib import contextmanager
@contextmanager
def fetch_certificate(url="", port=443, timeout=0.5):
    cxt = ssl.create_default_context()
    sslctxsock = cxt.wrap_socket(socket.socket(), server_hostname=hostname)
    sslctxsock.settimeout(timeout)
    sslctxsock.connect((url,port))
    cert = sslctxsock.getpeercert()
    try:
        yield cert
    finally:
        sslctxsock.close()

Use it like this:

with fetch_certificate(url=url) as cert:
   print(cert)

Suricate

Answer 2

After a number of attempts, some failed, some partial successful, I found a way that should work (didn't test it with self signed certificates, though). Also, I wiped out everything from the previous attempts.

There are 2 necessary steps:

1. Get the server certificate using [Python 3.Docs]: (ssl.get_server_certificate(addr, ssl_version=PROTOCOL_TLS, ca_certs=None), which returns it as a PEM encoded string (e.g.: ours - pretty printed):

   '-----BEGIN CERTIFICATE-----'
   'MIIIPjCCByagAwIBAgIICG/ofYt2G48wDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE'
   'BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl'
   
   ...
   
   'L2KuOvWZ40sTVCJdWPUMtT9VP7VHfLNTFft/IhR+bUPkr33xjOa0Idq6cL89oufn'
   '-----END CERTIFICATE-----'
   

2. Decode the certificate using (!!!undocumented!!!) ssl._ssl._test_decode_cert (present in Python 3 / Python 2)

3. Due to the fact that ssl._ssl._test_decode_cert can only read the certificate from a file, 2 additional steps are needed:

   • Save the certificate from #1. in a temporary file (before #2., obviously)

   • Delete that file when done with it

I would like to emphasize [Python 3.Docs]: SSLSocket.getpeercert(binary_form=False), which contains lots of info (that I missed the last time(s)).
Also, I found out about ssl._ssl._test_decode_cert, by looking at SSLSocket.getpeercert implementation ("${PYTHON_SRC_DIR}/Modules/_ssl.c").

code00.py:

#!/usr/bin/env python

import itertools
import os
import socket
import ssl
import sys


def _get_tmp_cert_file_name(host, port):
    return os.path.join(os.path.dirname(os.path.abspath(__file__)), "_".join(("cert", host, str(port), str(os.getpid()), ".crt")))


def _decode_cert(cert_pem, tmp_cert_file_name):
    #print(tmp_cert_file_name)
    with open(tmp_cert_file_name, "w") as fout:
        fout.write(cert_pem)
    try:
        return ssl._ssl._test_decode_cert(tmp_cert_file_name)
    except Exception as e:
        print("Error decoding certificate:", e)
        return dict()
    finally:
        os.unlink(tmp_cert_file_name)


def get_srv_cert_0(host, port=443):
    try:
        cert_pem = ssl.get_server_certificate((host, port))
    except Exception as e:
        print("Error getting certificate:", e)
        return dict()
    tmp_cert_file_name = _get_tmp_cert_file_name(host, port)
    return _decode_cert(cert_pem, tmp_cert_file_name)


def get_srv_cert_1(host, port=443):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    context = ssl.SSLContext()
    ssl_sock = context.wrap_socket(sock, server_hostname=host)
    try:
        ssl_sock.connect((host, port))
    except Exception as e:
        print("Error connecting:\n", e)
        return dict()
    try:
        cert_der = ssl_sock.getpeercert(True)
    except Exception as e:
        print("Error getting cert:\n", e)
        return dict()
    tmp_cert_file_name = _get_tmp_cert_file_name(host, port)
    return _decode_cert(ssl.DER_cert_to_PEM_cert(cert_der), tmp_cert_file_name)


def main(*argv):
    domain = "google.com"
    if argv:
        print("Using custom method")
        get_srv_cert_func = get_srv_cert_1
    else:
        print("Using regular method")
        get_srv_cert_func = get_srv_cert_0

    cert = get_srv_cert_func(domain)
    print("====== peer's certificate ======")
    try:
        print("Issued To:", dict(itertools.chain(*cert["subject"]))["commonName"])
        print("Issued By:", dict(itertools.chain(*cert["issuer"]))["commonName"])
        print("Valid From:", cert["notBefore"])
        print("Valid To:", cert["notAfter"])
        if (cert == None):
            print("no certificate")
    except Exception as e:
        print("Error getting certificate:", e)


if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Notes:

• _get_tmp_cert_file_name: generates the temporary file name (located in the same dir as the script) that will store the certificate

• _decode_cert: saves the certificate in the file, then decodes the file and returns the resulting dict

• get_srv_cert_0: gets the certificate form server, then decodes it

• get_srv_cert_1: same thing that get_srv_cert_0 does, but "manually"

  • Its advantage is controlling the SSL context creation / manipulation (which I think was the main point of the question)

• main:

  • Gets the server certificate using one of the 2 methods above (based on an argument being / not being passed to the script)

  • Prints certificate data (your code with some small corrections)

Output:

(py35x64_test) e:\Work\Dev\StackOverflow\q050055935> "e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" ./code00.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] 064bit on win32

Using regular method
====== peer's certificate ======
Issued To: *.google.com
Issued By: Google Internet Authority G2
Valid From: Apr 10 18:58:05 2018 GMT
Valid To: Jul  3 18:33:00 2018 GMT

Done.

(py35x64_test) e:\Work\Dev\StackOverflow\q050055935> "e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" ./code00.py 1
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] 064bit on win32

Using custom method
====== peer's certificate ======
Issued To: *.google.com
Issued By: Google Internet Authority G2
Valid From: Apr 10 18:55:13 2018 GMT
Valid To: Jul  3 18:33:00 2018 GMT

Done.

Check [SO]: How can I decode a SSL certificate using python? (@CristiFati's answer) for the decoding part only.

Selfabnegation

Thanks. Unfortunately this is not what I want. I need my python script to be able to trust or (unverified) my server's self-signed cert. How to do this? – Listerism

I see. Are you targeting your server only or any server in the world? – Selfabnegation

For now this server only. – Listerism

I have ithe server's certificate. But it will be helpful if you answer both cases. May be in future I need to consider any server's self-signed certificate. – Listerism

I posted a totally different solution. Hope it works for you. – Selfabnegation

I will try it but I find a much simpler way that did my purpose. I will post it also. – Listerism

FYI: I wrote the code as it is (with functions, error handling, and so on - which might give the false impression of complexity) so that it's modular and correct from the design's PoV. The cert decoding could be done in ~5 lines of code (if that means it's simpler). – Selfabnegation

This code is potentially dangerous since it's assuming get_server_certificate is talking to the right server (which you can't know unless you verify the cert--what the CA is for). Someone could MITM on that step and you'd get the wrong cert and blindly trust it. With that in mind, if you're trusting the network (generally not a good idea) you may as well just use HTTP and skip TLS – Homeopathy

Answer 3

It looks like Python doesn't parse/store the client's cert when ssl.CERT_NONE is set which is the default for SSLContext

import socket
import ssl

dest = ("www.google.com", 443)

sock = socket.socket(socket.AF_INET)
ctx = ssl.SSLContext()

# You will either need to add the self-signed certs to the OS cert store (varies by OS)
# See https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_verify_locations for loading non-defaults
ctx.set_default_verify_paths()

ctx.verify_mode = ssl.CERT_REQUIRED
ssock = ctx.wrap_socket(sock, server_hostname=dest[0])
result = ssock.connect(dest)
print(ssock.getpeercert())

See https://github.com/python/cpython/blob/main/Modules/_ssl.c#L1823 It's doing bitwise & so when verify mode is CERT_NONE (0) then the code block is skipped. CERT_OPTIONAL=1 or CERT_REQUIRED=2 is needed for the cert parsing/populating to work (otherwise it just sets an empty dict)

For your self-signed certs, you should either import them into the OS certificate authority store or use the method SSLContext.load_verify_locations to load the self-signed cert (since it's self-signed, you can load the server/leaf cert as the CA cert)

If you go the OS-store path, you can probably just use the default context. I think on Windows there's a per-user CA store (not positive, though). I don't think such a thing exists on Linux so you'd need to add it system-wide. Not sure about macOS or other operating systems

Homeopathy