Python - requests.exceptions.SSLError - dh key too small

Question

I'm scraping some internal pages using Python and requests. I've turned off SSL verifications and warnings.

requests.packages.urllib3.disable_warnings()
page = requests.get(url, verify=False)

On certain servers I receive an SSL error I can't get past.

Traceback (most recent call last):
  File "scraper.py", line 6, in <module>
    page = requests.get(url, verify=False)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: SSL_NEGATIVE_LENGTH] dh key too small (_ssl.c:600)

This happens both in/out of Cygwin, in Windows and OSX. My research hinted at outdated OpenSSL on the server. I'm looking for a fix client side ideally.

Edit: I was able to resolve this by using a cipher set

import requests

requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += 'HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST += 'HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

page = requests.get(url, verify=False)

Halloween

What kind of fix are you looking for? If the SSL cert has a problem, do you want to use HTTP or continue using HTTPS whilst ignoring the issue? – Intractable

@MarcelWilson: this is not a problem of the certificate. – Betelgeuse

@SteffenUllrich You're right of course. I should have stated if 'SSL' generically has a problem. – Intractable

move your code solution into the answer will help others to locate solution easily. – Fugitive

Answer 1

Disabling warnings or certificate validation will not help. The underlying problem is a weak DH key used by the server which can be misused in the Logjam Attack.

To work around this you need to chose a cipher which does not make any use of Diffie Hellman Key Exchange and thus is not affected by the weak DH key. And this cipher must be supported by the server. It is unknown what the server supports but you might try with the cipher AES128-SHA or a cipher set of HIGH:!DH:!aNULL

Using requests with your own cipher set is tricky. See Why does Python requests ignore the verify parameter? for an example.

Betelgeuse

Thanks for your suggestion, I was able to get past these errors using the cipher set and the post provided. – Halloween

Not very adept at this, how can I use your latter suggestion of trying another cipher with urllib2, please have a look at my issue #52440629 – Lampe

Another option (instead of telling the client to not use DH) is to configure the server to use more bits using dhparam see pointers at stackoverflow.com/a/64581683 – Prison

@sparrowt: the OP explicitly stated: "I'm looking for a fix client side ideally.": – Betelgeuse

Indeed hence not posting this as an answer - however people will find this question who do have server access, and in that case I thought it would be helpful to provide the context of how to fix it server side (as it would have been to me when I came here) rather than side-stepping it and leaving the server using an insecure DH config which other clients would then have to avoid too. – Prison

Answer 2

this is not an extra answer just try to combine the solution code from question with extra information So others can copy it directly without extra try

It is not only a DH Key issues in server side, but also lots of different libraries are mismatched in python modules.

Code segment below is used to ignore those securitry issues because it may be not able be solved in server side. For example if it is internal legacy server, no one wants to update it.

Besides the hacked string for 'HIGH:!DH:!aNULL', urllib3 module can be imported to disable the warning if it has

import requests
import urllib3

requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

page = requests.get(url, verify=False)

Fugitive

It's more secure to use custom HTTPAdapter that added the rule to whole Session, i.e. requests.Session().mount("affected.website", MySSLContextAdapter(ssl_ciphers='HIGH:!DH:!aNULL')) where that adapter creates context = ssl.create_default_context() and sets context.set_ciphers(self.ssl_ciphers) and uses it in HTTPAdapter.init_poolmanager. – Dicotyledon

If you're interacting with the socket directly, you can set these ciphers through the context (in Python 3.2+ and 2.7.9+): context.set_ciphers('HIGH:!DH:!aNULL') – Indoctrinate

You saved a business with this comment.. the bank we were using for payment had messed up their ssl this saved the day for now.. giving us enough time to change and work with another bank – Gust

I do not need 'verify=False'. it works. I want to say the hacked string for 'HIGH:!DH:!aNULL', does not have serious security issue for deprecation of ((Diffie Hellman)) in TLSv1.1 and TLSv1.2 – Acidulent

Thanks @Larry Cai! Your tip help me to do a POST method using Databricks with REST API. – Kotick

If you are using httpx you can do effectively the same thing with httpx._config.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL' – Keys

This works for urllib3<2. github.com/psf/requests/issues/6443#issuecomment-1535667256 – Verbena

Answer 3

This also worked for me:

import requests
import urllib3
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL:@SECLEVEL=1'

openssl SECLEVELs documentation: https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_security_level.html

SECLEVEL=2 is the openssl default nowadays, (at least on my setup: ubuntu 20.04, openssl 1.1.1f); SECLEVEL=1 lowers the bar.

Security levels are intended to avoid the complexity of tinkering with individual ciphers.

I believe most of us mere mortals don't have in depth knowledge of the security strength/weakness of individual ciphers, I surely don't. Security levels seem a nice method to keep some control over how far you are opening the security door.

Note: I got a different SSL error, WRONG_SIGNATURE_TYPE instead of SSL_NEGATIVE_LENGTH, but the underlying issue is the same.

Error:

Traceback (most recent call last):
  [...]
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='somehost.com', port=443): Max retries exceeded with url: myurl (Caused by SSLError(SSLError(1, '[SSL: WRONG_SIGNATURE_TYPE] wrong signature type (_ssl.c:1108)')))

Hentrich

Answer 4

I had the same issue.

And it was fixed by commenting

CipherString = DEFAULT@SECLEVEL=2

line in /etc/ssl/openssl.cnf .

Curhan

What i did was to set the 2 to 1 instead of commenting the hole line and it worked too... Better some level of security than none, right? – Berezina

RUN sed -i 's/DEFAULT@SECLEVEL=2/DEFAULT@SECLEVEL=1/' /etc/ssl/openssl.cnf – Agamogenesis

Answer 5

Starting from version 2 of urllib3, or version 2.30 of requests, the DEFAULT_CIPHERS attribute is removed and most proposed solutions here do not work anymore. Instead you'll have to create an SSLContext:

from urllib3.util import create_urllib3_context
from urllib3 import PoolManager

ctx = create_urllib3_context(ciphers=":HIGH:!DH:!aNULL")
http = PoolManager(ssl_context=ctx)
http.request("GET", ...)

If you're using requests you will have to subclass your own HTTPAdapter which initializes a PoolManager, and mount it into a Session:

from urllib3.util import create_urllib3_context
from urllib3 import PoolManager
from requests.adapters import HTTPAdapter
from requests import Session

class AddedCipherAdapter(HTTPAdapter):
  def init_poolmanager(self, connections, maxsize, block=False):
    ctx = create_urllib3_context(ciphers=":HIGH:!DH:!aNULL")
    self.poolmanager = PoolManager(
      num_pools=connections,
      maxsize=maxsize,
      block=block,
      ssl_context=ctx
    )

s = Session()
s.mount("https://example.org", AddedCipherAdapter())
s.get("https://example.org/path")

This approach has the benefit of being scoped only to the affected server, here https://example.org.

Jeopardous

This is the best answer! Worked for me after also setting verify=False on the session object (our server both had a bad key AND failed to authenticate). – Dilisio

Cannot set verify_mode to CERT_NONE when check_hostname is enabled. – Brewington

@Mirko, add ctx.check_hostname = False after ctx = create_urllib3_context – Nephrectomy

Answer 6

Someone from the requests python library's core development team has documented a recipe to keep the changes limited to one or a few servers:

https://lukasa.co.uk/2017/02/Configuring_TLS_With_Requests/

If your code interacts with multiple servers, it makes sense not to lower the security requirements of all connections because one server has a problematic configuration.

The code worked for me out of the box. That is, using my own value for CIPHERS, 'ALL:@SECLEVEL=1'.

Hentrich

Answer 7

It's may be safer not to override the default global ciphers, but instead create custom HTTPAdapter with the required ciphers in a specific session:

import ssl
from typing import Any

import requests

class ContextAdapter(requests.adapters.HTTPAdapter):
    """Allows to override the default context."""

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        self.ssl_context: ssl.SSLContext|None = kwargs.pop("ssl_context", None)

        super().__init__(*args, **kwargs)

    def init_poolmanager(self, *args: Any, **kwargs: Any) -> Any:
        # See available keys in urllib3.poolmanager.SSL_KEYWORDS
        kwargs.setdefault("ssl_context", self.ssl_context)

        return super().init_poolmanager(*args, **kwargs)

then you need to create custom context, for example:

import ssl

def create_context(
    ciphers: str, minimum_version: int, verify: bool
) -> ssl.SSLContext:
    """See https://peps.python.org/pep-0543/."""
    ctx = ssl.create_default_context()

    # Allow to use untrusted certificates.
    if not verify:
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE

    # Just for example.
    if minimum_version == ssl.TLSVersion.TLSv1:
        ctx.options &= (
            ~getattr(ssl, "OP_NO_TLSv1_3", 0)
            & ~ssl.OP_NO_TLSv1_2
            & ~ssl.OP_NO_TLSv1_1
        )
        ctx.minimum_version = minimum_version

    ctx.set_ciphers(ciphers)

    return ctx

and then you need to configure each website with custom context rules:

session = requests.Session()
session.mount(
    "https://dh.affected-website.com",
    ContextAdapter(
        ssl_context=create_context(
            ciphers="HIGH:!DH:!aNULL"
        ),
    ),
)
session.mount(
    "https://only-elliptic.modern-website.com",
    ContextAdapter(
        ssl_context=create_context(
            ciphers="ECDHE+AESGCM"
        ),
    ),
)
session.mount(
    "https://only-tls-v1.old-website.com",
    ContextAdapter(
        ssl_context=create_context(
            ciphers="DEFAULT:@SECLEVEL=1",
            minimum_version=ssl.TLSVersion.TLSv1,
        ),
    ),
)

result = session.get("https://only-tls-v1.old-website.com/object")

After reading all the answers, I can say that @bgoeman's answer is close to mine, you can follow their link to learn more.

Dicotyledon

Just be advised that if you're using pip-system-certs, it hot-patches the HTTPAdapter base class, meaning super().init_poolmanager won't behave as you expect. I found this out the hard way. – Enalda

Answer 8

Based on the answer given by the user bgoeman, the following code, which keeps the default ciphers only adding the security level, works.

import requests
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += '@SECLEVEL=1'

Aesop

Answer 9

I will package my solution here. I had to modify the python SSL library, which was possible since I was running my code within a docker container, but it's something that probably you don't want to do.

1. Get the supported cipher of your server. In my case was a third party e-mail server, and I used script described list SSL/TLS cipher suite

check_supported_ciphers.sh

#!/usr/bin/env bash

# OpenSSL requires the port number.
SERVER=$1
DELAY=1
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')

echo Obtaining cipher list from $(openssl version).

for cipher in ${ciphers[@]}
do
echo -n Testing $cipher...
result=$(echo -n | openssl s_client -cipher "$cipher" -connect $SERVER 2>&1)
if [[ "$result" =~ ":error:" ]] ; then
  error=$(echo -n $result | cut -d':' -f6)
  echo NO \($error\)
else
  if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher    :" ]] ; then
    echo YES
  else
    echo UNKNOWN RESPONSE
    echo $result
  fi
fi
sleep $DELAY
done

Give it permissions:

chmod +x check_supported_ciphers.sh

And execute it:

./check_supported_ciphers.sh myremoteserver.example.com | grep YES

After some seconds you will see an output similar to:

Testing AES128-SHA...YES (AES128-SHA_set_cipher_list)

So will use "AES128-SHA" as SSL cipher.

2. Force the error in your code:

   Traceback (most recent call last): File "my_custom_script.py", line 52, in imap = IMAP4_SSL(imap_host) File "/usr/lib/python2.7/imaplib.py", line 1169, in init IMAP4.init(self, host, port) File "/usr/lib/python2.7/imaplib.py", line 174, in init self.open(host, port) File "/usr/lib/python2.7/imaplib.py", line 1181, in open self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile) File "/usr/lib/python2.7/ssl.py", line 931, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 599, in init self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:727)

3. Get the python SSL library path used, in this case:

   /usr/lib/python2.7/ssl.py

4. Edit it:

   cp /usr/lib/python2.7/ssl.py /usr/lib/python2.7/ssl.py.bak

   vim /usr/lib/python2.7/ssl.py

And replace:

_DEFAULT_CIPHERS = (
    'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:'
    'ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:RSA+AESGCM:RSA+AES:RSA+HIGH:'
    '!aNULL:!eNULL:!MD5:!3DES'
    )

By:

_DEFAULT_CIPHERS = (
    'AES128-SHA'
    )

Gravure

No "YES" at all – Ague

Answer 10

I encounter this problem afer upgrading to Ubuntu 20.04 from 18.04, following command works for me .

pip install --ignore-installed pyOpenSSL --upgrade

Dacoit

I have Ubuntu 20.04 but it does not help me. – Acidulent

it messed up all of my existing python packages, and unable to install any package – Denude

Answer 11

On CentOS 7, search for the following content in /etc/pki/tls/openssl.cnf:

[ crypto_policy ]
.include /etc/crypto-policies/back-ends/opensslcnf.config  
[ new_oids ]  

Set 'ALL:@SECLEVEL=1' in /etc/crypto-policies/back-ends/opensslcnf.config.

Kisumu

Answer 12

In docker image you can add the following command in your Dockerfile to get rid of this issue:

RUN sed -i '/CipherString = DEFAULT/s/^#\?/#/' /etc/ssl/openssl.cnf

This automatically comments out the problematic CipherString line.

Monies

Answer 13

If you are using httpx library, with this you skip the warning:

import httpx

httpx._config.DEFAULT_CIPHERS += ":HIGH:!DH:!aNULL"

Duisburg

Answer 14

I had the following error:

• SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:727)

I solved it(Fedora):

• python2.7 -m pip uninstall requests

• python2.7 -m pip uninstall pyopenssl

• python2.7 -m pip install pyopenssl==yourversion

• python2.7 -m pip install requests==yourversion

The order module install cause that:

• requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS

AttributeError "pyopenssl" in "requests.packages.urllib3.contrib" when the module did exist.

Opulent

Answer 15

I've made an account just for this thread cause none of these answers worked for me. Using Robin De Schepper's piece of code from THIS comment as a baseline I have managed to make it work with only one small adjustment

Original piece of code:

from urllib3.util import create_urllib3_context
from urllib3 import PoolManager

ctx = create_urllib3_context(ciphers=":HIGH:!DH:!aNULL")
http = PoolManager(ssl_context=ctx)
http.request("GET", ...)

Modified piece of code:

import ssl # needed for the cert argument
from urllib3.util import create_urllib3_context
from urllib3 import PoolManager

# cert_reqs=ssl.CERT_NONE is the one adjustment that made this work
ctx = create_urllib3_context(ciphers=":HIGH:!DH:!aNULL", cert_reqs=ssl.CERT_NONE)
http = PoolManager(ssl_context=ctx)
http.request("GET", ...)

Bernhardt

Answer 16

I'll add this in case someone is having this exact problem with ftplib.

The solution is to create a context using the older configuration:

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
context.set_ciphers('DEFAULT@SECLEVEL=1')

with ftplib.FTP_TLS(
    host=self.ftp_host,
    user=self.ftp_user,
    passwd=self.ftp_password,
    context=context,
) as ftp:
    ...

Engage