How to disable hostname checking in requests python

Question

I'm using Requests to connect to a RESTful API. The server I would like to reach use ssl with self-singed certificate.

cafile = "gateway.pem"
r = requests.get(request, auth=('admin', 'password'), verify=cafile)

the problem is I'm getting SSLError of hostname mismatch. there should be a way to disable the hostname checking without disabling certificate validation, as in many java implementations, but I can't find how to do it with requests in python.

stacktrace:

Traceback (most recent call last):
  File "<pyshell#43>", line 1, in <module>
    r = requests.get(request, auth=("admin", "password"), verify='gateway.pem')
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests-2.0.0-py2.7.egg\requests\adapters.py", line 358, in send
    raise SSLError(e)
SSLError: hostname '10.76.92.70' doesn't match u'lital.com'

How can this be done?

Buote

Answer 1

Requests doesn't allow this directly, however you can provide a custom transport adapter which uses the features of the underlying urllib3. The usage of transport adapters is covered in the requests documentation.

This code is not tested, but should work.

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager


# Never check any hostnames
class HostNameIgnoringAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       assert_hostname=False)


# Check a custom hostname
class CustomHostNameCheckingAdapter(HTTPAdapter):
    def cert_verify(self, conn, url, verify, cert):
        #      implement me
        host = custom_function_mapping_url_to_hostname(url)
        conn.assert_hostname = host
        return super(CustomHostNameCheckingAdapter,
                     self).cert_verify(conn, url, verify, cert)

In detail the assert_hostname param works as follows: If None use the hostname from the URL, if False suppress hostname checking, if a custom string validate against this string.

Minimalist

Thanks for that. I actually tried this as you can see below, but found out that after changing to use my costume adapter, no certificate varification is performed at all (instead of only hostname verification). I passed requests a corrupted pem file and got valid response. – Buote

class MyAdapter(requests.adapters.HTTPAdapter): def init_poolmanager(self, connections, maxsize, block): self._pool_connections = connections self._pool_maxsize = maxsize self._pool_block = block self.poolmanager = requests.packages.urllib3.PoolManager(num_pools=connections, maxsize=maxsize, block=block, assert_hostname=False) – Buote

I'm then doing this: s = requests.Session() s.mount('https://', MyAdapter()) r = s.get(request, auth=("admin", "admin"), verify=cafile) – Buote

Any idea what is worng here and why this disables the certificate validation? – Buote

If I use a corrupted pem file openssl blows up for me requests.exceptions.SSLError: [Errno 0] _ssl.c:343: error:00000000:lib(0):func(0):reason(0) – Minimalist

Answer 2

I'm a little late to the party but requests_toolbelt looks like it might help if you install version 0.7.0 or newer (my ubuntu 16.04 only has 0.6.0): https://toolbelt.readthedocs.io/en/latest/adapters.html#hostheaderssladapter

From the link:

Example usage:
>>> s.mount('https://', HostHeaderSSLAdapter())
>>> s.get("https://93.184.216.34", headers={"Host": "example.org"})

Thier

Answer 3

Did you look into the SSLContext.check_hostname parameter? You should be able to set it to False, and it should not check the hostname:

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = True
context.load_default_certs()

The only limitation is that this only works in Python 3.4 and later.

Reference: https://docs.python.org/3/library/ssl.html#ssl.SSLContext.check_hostname

Foti

Answer 4

It looks like this feature has been added to the latest version of requests. I've verified and it's working for me - just pass verify=False on the request - as shown in the example in the documentation:

requests.get('https://api.github.com', verify=False)

Precincts

The problem with this approach is that it turns off certificate validation completely, it doesn't just turn off the hostname check. In the question, the desired check is for "the presented certificate is signed by the certificate given in cafile", which means that certificate verification needs to remain enabled at the requests level. – Croft

Does not works, the verify argument can not disable hostname validation. – Omnifarious

Answer 5

http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification

verify keyword is a flag, not for providing certfile. You provided there non-empy string, which resolves to True in boolean context.

Use cert= keyword to provide path to certificate files, or disable verification with verify=False.

EDIT: although documentation says you can actually pass CA path to verify=, but there are no examples. It would be helpful to see whole traceback you are receiving.

Stardom

When I'm doing the same call with verify=True I'm getting different error of "certificate verify failed". So looks like it is possible to pass the pem file to verify=. when I'm doing the call with cert=cafile instead of verify=cafile, I'm getting SSLError: [Errno 336265225] _ssl.c:351: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib. I added the traceback to the message. – Buote

what is your request? Looks like CA is hooked up properly, but it is signed for 'lital.com' and you connect by IP. Hence errors. – Stardom

right, the hostname in the certificate ('lital.com') isn't really the server's hostname. That's the reason I want to disable the hostname matching/checking. In java this can be done with ALLOW_ALL_HOSTNAME_VERIFIER. I couldn't find something similar in requests. – Buote

I don't want to disable entirely the certificate validation, only the hostname checking. – Buote

OK, now I get it. This is optional client feature as [docs say](rfc6125) and it may be unsupported in request, especially since it's security hazard. You can try: and if you catch this particular error, do manual check and if all seems ok for you, re-request with verify=False. Nothing else comes to my mind. – Stardom

PS. it doesn't look like it is supported: github.com/kennethreitz/requests/blob/master/requests/… but this may help you: #18062140 – Stardom

Answer 6

If this is only for testing only then just add an entry in /etc/hosts to your local system (assuming you have access).

Designation