Custom urllib opener that uses client certificates

Question

I have got an API that I have to work with. The API is secured by HTTPS and uses mutual authentication/client certificates. I have a PEM file and a CRT file.

When I connect to the server regularly, using PyOpenSSL I have no problem, here is the code:

import settings
from OpenSSL import SSL
import socket

def verify(conn, cert, errnum, depth, ok):
    # This obviously has to be updated
    print 'Got certificate: %s' % cert.get_subject()
    return ok

def password_callback(maxlen, verify, extra):
        print (maxlen, verify, extra)
        return settings.DEPOSIT_CODE

context = SSL.Context(SSL.SSLv23_METHOD)
context.set_verify(SSL.VERIFY_NONE, verify)
context.set_passwd_cb(password_callback)
context.use_certificate_file(settings.CLIENT_CERT_FILE)
context.use_privatekey_file(settings.PEM_FILE)

sock = SSL.Connection(context, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
sock.connect(("someserver.com",443))

http_get_request = """
GET / HTTP/1.1

"""
sock.write(http_get_request)
print sock.recv(1000)

But, because this is an HTTPS API with client certificate, I have implemented an opener for it, the code somehow modified is here:

import settings
import socket
import urllib2

def verify(conn, cert, errnum, depth, ok):
    # This obviously has to be updated
    print 'Got certificate: %s' % cert.get_subject()
    return ok

def password_callback(maxlen, verify, extra):
        print (maxlen, verify, extra)
        return settings.DEPOSIT_CODE

class MyHTTPSConnection(httplib.HTTPSConnection):
    def connect(self):
        context = SSL.Context(SSL.SSLv23_METHOD)
        context.set_passwd_cb(password_callback)
        context.use_certificate_file(settings.CLIENT_CERT_FILE)
        context.set_verify(SSL.VERIFY_NONE, verify)
        context.use_privatekey_file(settings.PEM_FILE)
        self.sock = SSL.Connection(context, socket.socket(socket.AF_INET, socket.SOCK_STREAM))

class MyHTTPSHandler(urllib2.HTTPSHandler):
    def https_open(self,req):
        return self.do_open(MyHTTPSConnection,req)

opener = urllib2.build_opener(urllib2.HTTPHandler,MyCHTTPSHandler)
urllib2.install_opener(opener)

f = urllib2.urlopen("https://sampleapiserver.com")
print f.code

but when I run the second code, I get the following error:

  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "network.py", line 37, in https_open
    return self.do_open(IRNICHTTPSConnection,req)
  File "/usr/lib/python2.6/urllib2.py", line 1142, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.6/httplib.py", line 914, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.6/httplib.py", line 951, in _send_request
    self.endheaders()
  File "/usr/lib/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib/python2.6/httplib.py", line 759, in send
    self.sock.sendall(str)
OpenSSL.SSL.Error: [('SSL routines', 'SSL_write', 'uninitialized')]

Finally, am I doing something wrong? If not, please help me understand the error...

Cheers.

Doble

Could you please clarify the part where you say this is an HTTP API, but you want to get rid of the HTTP protocol? – Kithara

Answer 1

I'm not sure - but it looks to me like you are missing doing the connect() call in the connect() method:

self.sock.connect(("someserver.com",443))

Also httplib's https handling has wrapper classes for the SSL socket, so maybe those are required for it to work?

Crosspatch

Yeah, it is exactly what I did yesterday and had it fixed :) – Doble

Answer 2

It looks like you're adding a lot of complexity here that you don't really need. If you're just doing simple client certificate authentication, you could probably get away from the following snippet (source):

import httplib
import urllib2

# HTTPS Client Auth solution for urllib2, inspired by
# http://bugs.python.org/issue3466
# and improved by David Norton of Three Pillar Software. In this
# implementation, we use properties passed in rather than static module
# fields.
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
    def __init__(self, key, cert):
        urllib2.HTTPSHandler.__init__(self)
        self.key = key
        self.cert = cert
    def https_open(self, req):
        #Rather than pass in a reference to a connection class, we pass in
        # a reference to a function which, for all intents and purposes,
        # will behave as a constructor
        return self.do_open(self.getConnection, req)
    def getConnection(self, host):
        return httplib.HTTPSConnection(host, key_file=self.key, cert_file=self.cert)


cert_handler = HTTPSClientAuthHandler(settings.PEMFILE, settings.CLIENT_CERT_FILE)
opener = urllib2.build_opener(cert_handler)
urllib2.install_opener(opener)

f = urllib2.urlopen("https://sampleapiserver.com")
print f.code

The source was used in the context of providing a cert-authenicated URL opener to the Suds Client constructor, so I stripped that out and made it a direct opener.

Hindi

The only problem with this is that you can't set a password callback (Python's built in SSL support did not have this facility until version 3.3 or so), so if the client key is password-protected, there is no way to prevent the underlying SSL library from trying to prompt the user on the terminal. – Perugia