Python SSL certificate verify error
Asked Answered
M

7

86

I'm using requests to access a RESTful API. Everything seems to work. I can authenticate, pull back a session token and even unit test the methods in my class I wrote for the API. Then I tried to run my code.

First, here is the call I'm making. The headers are static session-related items that get set in init(). The body is build dynamically from data in a file and passed in to this function. All of the data is valid.

response = requests.post(url, headers=(Requestheader), data=json.dumps((Requestbody)))

When I run the code, it updates well over 100 records with the metadata I supply. Somewhere around item 150 I get the following:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)

My first step was to call the vendor and find out if ALL of their web servers had properly signed certs figuring they were load balancing me and I found a misconfigured server. They tell me this is not the case.

Then I Googled the message and found that there is a verify kwarg, so I tried:

response = requests.post(url, headers=Requestheader, data=json.dumps(Requestbody), verify=False)

I know this isn't ideal long-term, but I wanted to test it to see if The behavior is the same. It did the same thing. It ran for a while and threw the ssl error. I thought the idea of verify=False is that it wouldn't check.

The vendor suggested I check the url I'm using, but it's fine. I would think if there were a proxy server or real man-in-the-middle attack causing problems I wouldn't see success so many times before a failure. I thought maybe it is a session timeout, but that should throw a 401 status and my activity level is too high for an inactivity timeout.

I'm a python noob and not a security professional. Suggestions appreciated.

Marleah answered 17/7, 2018 at 22:23 Comment(1)
Which version of requests? You should try to add more debug or capture the traffic to see each certificate you get, and see what changes. verify=False is in all cases not something that should be kept.Apparatus
I
18

So the issue might have three resolutions as I see it:

  1. A certificate is OK and there is something wrong with the code. The issue may occur, for example, while using prepared requests as described in this solution

    But I don't really think it is your case because in the snippet you've provided no such methods are used. For the two next variants, you'll need to get the URL that causes an error and explore it's certificate (can be done via browser).

  2. A certificate is OK but a certificate authority that signed it is not included in CA list that is utilized by requests library. After you'll open a troubling URL, check CA in it and see if it's dates are valid and it is included in this list. If not, add CA in the trusted list for the requests library -- as explained in the answers to this StackOverflow question.

  3. A certificate is not valid or self-singed. Same solution as in 2.

The general solution is to wrap your script in the try except clause and to print out all the URLs that will result in mistakes. Then try to request them one by one via requests library and see if the issue occurs. If it does, it's (2) or (3) case. If not — try to run script on another machine with fresh installed python and requests. If the run will be successful — then there's some issue in your configuration.

Inellineloquent answered 18/7, 2018 at 18:45 Comment(1)
Thank you for the comprehensive response. I learned a few things. 1) I should be using sessions since I call a number of APIs at the site 2) It made me look again at the cert chain and it looks like there are two paths. On the second path there is a self-signed cert. ssllabs.com/ssltest/analyze.html?d=api.riscnetworks.com I'm going to look at your first link again on how to add the trust.Marleah
D
76

Update

python-certifi-win32 is not maintained anymore as pointed out in the comment.

Use the drop-in library pip-system-certs instead.

pip install pip-system-certs

OBSOLETE

If you are using Windows and you have already imported the CA in the trusted DB store, you can install the package python-certifi-win32 that automagically will use the same certificates from the Trusted DB store.

pip install python-certifi-win32

Then your code should start to work

See: Python Requests with wincertstore

https://gitlab.com/alelec/python-certifi-win32

Dasyure answered 23/1, 2021 at 14:30 Comment(3)
Unfortunately python-certifi-win32 seems to be out-of-maintenance. Andrew Leech, the original author, suggests to move on to his other project pip-system-certs. Drop-in replacing the library with pip-system-certs (PyPI link) seems to work flawlessly.Kawai
pip install pip-system-certs worked for me => I would kindly suggest to update the post to make it more clearModlin
I was trying unsuccessfully from python to use the cert parameter in requests calls. When I tried this solution, I did not use the cert parameter (kwarg), and this fixed my request issues. Many thanks for the answer AND for keeping it updated.Captor
S
39

I have found this over here

I found this solution, insert this code at the beginning of your source file:

import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context
Significancy answered 24/3, 2019 at 5:24 Comment(2)
This is not advisable. requests has a verify=False option that can turn off SSL verification. However, that means that you are no longer checking that the machine you are talking with is the one you want to talk with. It leaves you open to a man in the middle attack. My final workaround was to add a retry decorator on all of my functions calling the vendor API.Marleah
Since I was inside docker talking from one container to another and didn't wanna go into adding a CA myself valid on both containers, this was exactly the solution.Numbers
I
18

So the issue might have three resolutions as I see it:

  1. A certificate is OK and there is something wrong with the code. The issue may occur, for example, while using prepared requests as described in this solution

    But I don't really think it is your case because in the snippet you've provided no such methods are used. For the two next variants, you'll need to get the URL that causes an error and explore it's certificate (can be done via browser).

  2. A certificate is OK but a certificate authority that signed it is not included in CA list that is utilized by requests library. After you'll open a troubling URL, check CA in it and see if it's dates are valid and it is included in this list. If not, add CA in the trusted list for the requests library -- as explained in the answers to this StackOverflow question.

  3. A certificate is not valid or self-singed. Same solution as in 2.

The general solution is to wrap your script in the try except clause and to print out all the URLs that will result in mistakes. Then try to request them one by one via requests library and see if the issue occurs. If it does, it's (2) or (3) case. If not — try to run script on another machine with fresh installed python and requests. If the run will be successful — then there's some issue in your configuration.

Inellineloquent answered 18/7, 2018 at 18:45 Comment(1)
Thank you for the comprehensive response. I learned a few things. 1) I should be using sessions since I call a number of APIs at the site 2) It made me look again at the cert chain and it looks like there are two paths. On the second path there is a self-signed cert. ssllabs.com/ssltest/analyze.html?d=api.riscnetworks.com I'm going to look at your first link again on how to add the trust.Marleah
R
15

For me it was sufficient just to add verify=False to the request:

response = requests.request("POST", url, headers=headers, data=payload, verify=False)

Hopefully this very easy answer helps anybody.

Retrogressive answered 24/6, 2022 at 8:43 Comment(2)
Tkx bro. It solves for me too.Puebla
This means that the SSL certificate of the HTTPS server being accessed will not be checked, and any HTTPS server (even a malicious one) could present a self-signed or invalid SSL certificate without the client being warned.Stricken
C
10

For Debian:

Download certificate

openssl s_client -showcerts -connect SERVER:443 </dev/null 2>/dev/null | sed -n -e '/BEGIN\ CERTIFICATE/,/END
CERTIFICATE/ p' > git-mycompany-com.pem

Install

pip install certifi

Execute python

Python 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import certifi

certifi.where() '/home/user/detectaEnv/lib/python3.7/site-packages/certifi/cacert.pem'

exit()

Execute command for add perm to python

cat git-mycompany-com.pem | tee -a /home/user/detectaEnv/lib/python3.7/site-packages/certifi/cacert.pem

It has worked perfectly for me

Crissum answered 9/11, 2021 at 13:4 Comment(1)
This is the right way. I have verified it with mitmproxy and conda Python 3.10 on Ubuntu 22.04.Huntington
H
4

You can set the resource URL as a trusted host by executing the below command:

In this case i have installed the package jupyterlab

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org jupyterlab
Hutment answered 5/11, 2021 at 10:0 Comment(0)
P
0

Not very advisable, but a temporary workaround can be to downgrade to requests==2.21.0 and then to place this at the top of your script:

import os 
os.environ['CURL_CA_BUNDLE'] = ''

You will get the following warnings:

Unverified HTTPS request is being made. Adding certificate verification is strongly advised.

I am only sharing this because it may be useful to some people! (it was for me). This is a slightly different workaround to that suggested by @user1141785 but it can be used in cases where the request is not being explicitely called in your code, but instead through some other exposed function.

Pyuria answered 17/4, 2024 at 7:9 Comment(0)

© 2022 - 2025 — McMap. All rights reserved.