python ignore certificate validation urllib2

Question

I want to ignore the certification validation during my request to the server with an internal corporate link.

With python requests library I would do this:

r = requests.get(link, allow_redirects=False,verify=False)

How do I do the same with urllib2 library?

Equilateral

Answer 1

urllib2 does not verify server certificate by default. Check this documentation.

Edit: As pointed out in below comment, this is not true anymore for newer versions (seems like >= 2.7.9) of Python. Refer the below ANSWER

Tuantuareg

This doesn't seem to be true anymore. – Buckingham

Indeed that's not true anymore. – Laroche

so why is he being downvoted if his answre was correct at the time? have some class – Extrinsic

"not true anymore" - that's utterly useless. The world didn't all get their Python versions magically upgraded - it is actually still true for anyone still on 2.7.6! – Nananne

Answer 2

In the meantime urllib2 seems to verify server certificates by default. The warning, that was shown in the past disappeared for 2.7.9 and I currently ran into this problem in a test environment with a self signed certificate (and Python 2.7.9).

My evil workaround (don't do this in production!):

import urllib2
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

urllib2.urlopen("https://your-test-server.local", context=ctx)

According to docs calling SSLContext constructor directly should work, too. I haven't tried that.

Binkley

so evil and so working xD thanks for that. I already were fallen into despair. – Diplomatist

It seems that ssl.create_default_context is only available in Python 3.4+. – Kaczmarek

This works for me using python 2.7.9 (OSX Homebrew installed) Thanks! – Hatchett

This workaround doesn't work with either python 2.6.6 or 2.7.6. – Laroche

This work-around is perfect, but just don't call urllib2.urlopen() prior to 2.7.9 with the context parameter, that's all. I'm using this in both 2.7.10 and 2.6.x. Check version using this code: sys.version_info >= ( 2, 7, 9 ). – Mccubbin

So... this was about to work for me, but I've also got to supply HTTP Basic authentication information at the same time. This technique seems to interfere with using urllib2.install_opener(urllib2.build_opened(urllib2.HTTPBasicAuthHandler(mypwdmgr)). Any suggestions for getting these working together? – Ultraviolet

@ChristopherSchultz pass urllib2.HTTPSHandler(context=ctx) as additional argument to urllib2.build_opener – Exciting

@Exciting Yeah that's as clear as mud from the Python documentation, but I did in fact get it figured out a while back (sorry I didn't post back with an update). Since I needed to support optional authentication and/or hostname-verification-skipping, I ended up with a few nested conditionals to figure out what was happening and then calling urllib2.install_opener(urllib2.build_opener(auth_handler, host_handler)) when I needed both. – Ultraviolet

@eightx2 Do you know if there's anything like that on Python2.7? – Plantaineater

You say not to use this in production - what is it exactly that makes ignoring the cert validation so dangerous? Is it still dangerous if you know the URL you're hitting is safe? (e.g., in the original question the OP said they're hitting an internal corporate link, which will always be safe) – Dolce

I don't see the point of using encryption without verification. If you fully trust the network, why are you using encryption? In security I always think of the onion layer model. Every layer improves security and makes you depend less on the other layers. If you don't verify the endpoint, encryption is only producing heat. In my opinion most attackers, that can read your packets may probably modify them too. – Buckingham

This worked perfectly in python 2.7.13 using urllib (not urllib2, which I didn't try) ssl.create_default_context was in fact available for me. – Dochandorrach

Answer 3

The easiest way:

python 2

import urllib2, ssl

request = urllib2.Request('https://somedomain.co/')
response = urllib2.urlopen(request, context=ssl._create_unverified_context())

python 3

from urllib.request import urlopen
import ssl

response = urlopen('https://somedomain.co', context=ssl._create_unverified_context())

Seventeen

Make sure you have a later python version. The version shipped with ubuntu 14.04 does not support this method. – Interjoin

This method works but had change import statement to from urllib.request import urlopen instead of import urllib2. See the accepted answer at #2793150 for more info. – Scribble

Answer 4

For those who uses an opener, you can achieve the same thing based on Enno Gröper's great answer:

import urllib2, ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx), your_first_handler, your_second_handler[...])
opener.addheaders = [('Referer', 'http://example.org/blah.html')]

content = opener.open("https://localhost/").read()

And then use it as before.

According to build_opener and HTTPSHandler, a HTTPSHandler is added if ssl module exists, here we just specify our own instead of the default one.

Susuable

Good call, this looks similar to the method described at this site. thejosephturner.com/blog/post/… – Heirdom

I disagree, IMHO, it's not similar. The article you mention uses a class extention while this method uses vanilla implementation and only ssl context. That said, the article you mentioned is a good point for whoever wants to alter and extend default HTTPSHandler behavior. – Susuable

I see what you mean, I hadn't caught that nuance, yours is definitely cleaner without having to extend anything. – Heirdom

Answer 5

According to @Enno Gröper 's post, I've tried the SSLContext constructor and it works well on my machine. code as below:

import ssl
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
urllib2.urlopen("https://your-test-server.local", context=ctx)

if you need opener, just added this context like:

opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx))

NOTE: all above test environment is python 2.7.12. I use PROTOCOL_SSLv23 here since the doc says so, other protocol might also works but depends on your machine and remote server, please check the doc for detail.

Baldpate

Answer 6

A more explicit example, built on Damien's code (calls a test resource at http://httpbin.org/). For python3. Note that if the server redirects to another URL, uri in add_password has to contain the new root URL (it's possible to pass a list of URLs, also).

import ssl    
import urllib.parse
import urllib.request

def get_resource(uri, user, passwd=False):
    """
    Get the content of the SSL page.
    """
    uri = 'https://httpbin.org/basic-auth/user/passwd'
    user = 'user'
    passwd = 'passwd'

    context = ssl.create_default_context()
    context.check_hostname = False
    context.verify_mode = ssl.CERT_NONE

    password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
    password_mgr.add_password(None, uri, user, passwd)

    auth_handler = urllib.request.HTTPBasicAuthHandler(password_mgr)

    opener = urllib.request.build_opener(auth_handler, urllib.request.HTTPSHandler(context=context))

    urllib.request.install_opener(opener)

    return urllib.request.urlopen(uri).read()

Kattiekatuscha

Thanks for answering the python3 way with standard library. Preparing context worked for me very well. – Peaceful