MD4 hashlib support in Python 3.8
Asked Answered
T

5

10

I am trying to implement a soap client for a server that uses NTLM authentication. The libraries that I use (requests-ntlm2 which relies on ntlm-auth) implement the MD4 algorithm that lies in the core of the NTLM protocol via the standard library's hashlib.

Although hashlib seems to support MD4:

>>> import hashlib
>>> hashlib.algorithms_available
{'md5-sha1', 'md4', 'shake_128', 'md5', 'blake2s', 'sha3_512', 'ripemd160', 'sha512', 'mdc2', 'blake2b', 'sha3_256', 'sha3_224', 'sha512_224', 'sha1', 'sha384', 'sha256', 'sha224', 'whirlpool', 'sha512_256', 'sha3_384', 'shake_256', 'sm3'}
>>>

and so does the openssl library in my system:

(victory) C:\code\python\services>openssl
help:
[...]
Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        md4               md5
mdc2              rmd160            sha1              sha224
sha256            sha3-224          sha3-256          sha3-384
sha3-512          sha384            sha512            sha512-224
sha512-256        shake128          shake256          sm3
[...]

when the authentication tries to run python produces an ValueError: unsupported hash type md4 error. Here is the relevant part of the traceback:

C:\ProgramData\Miniconda3\envs\victory\lib\site-packages\ntlm_auth\compute_hash.py in _ntowfv1(password)
    165         return nt_hash
    166 
--> 167     digest = hashlib.new('md4', password.encode('utf-16-le')).digest()
    168 
    169     return digest

C:\ProgramData\Miniconda3\envs\victory\lib\hashlib.py in __hash_new(name, data, **kwargs)
    161         # This allows for SHA224/256 and SHA384/512 support even though
    162         # the OpenSSL library prior to 0.9.8 doesn't provide them.
--> 163         return __get_builtin_constructor(name)(data)
    164 
    165 

C:\ProgramData\Miniconda3\envs\victory\lib\hashlib.py in __get_builtin_constructor(name)
    118         return constructor
    119 
--> 120     raise ValueError('unsupported hash type ' + name)
    121 
    122 

ValueError: unsupported hash type md4

Even when I try to merely call the MD4 from hashlib, I get the same result:

>>> import hashlib
>>> hashlib.new('md4')
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\victory\lib\hashlib.py", line 157, in __hash_new
    return _hashlib.new(name, data)
ValueError: [digital envelope routines] initialization error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Miniconda3\envs\victory\lib\hashlib.py", line 163, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "C:\ProgramData\Miniconda3\envs\victory\lib\hashlib.py", line 120, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md4

Any insights about what's going on and/or any help would be immensely appreciated.

Tan answered 12/11, 2021 at 6:1 Comment(7)
Following. This works fine for me.Coons
@FrankYellin are you using the same stack (windows, python 3.8, requests-ntlm2)? hashlib.new('md4') works allright?Tan
I have no specialized knowledge to add, but I'm on Windows, with a fresh Python 3.9, no OpenSSL available via PATH, and hashlib.new('md4') works for me.Suborder
If nothing else, there are pure Python implementations of md4. gist.github.com/kangtastic/c3349fc4f9d659ee362b12d7d8c639b6Coons
What version of openssl are you using? I had a similar issue, getting the same error message for code that had previously worked. After a lot of searching online for a solution, I finally figured out the culprit was the version of openssl being used. I rebuild my environment and explicitly specified openssl=1.1.1 - this solved the problem for me.Brewer
I have this problem with openssl 3.x on Windows. If I revert back to 1.1.1 it worksPromiscuous
@MarkThomas is it possible to create a virtual environment with customized packages? If I try to create a virtual environment with python 3.7, in packages I get openssl 3.x by default.Ishmul
T
2

Well, it seems that there was something corrupted in my conda environment. I created a new identical one, and it's been working ever since without having to change anything else.

Tan answered 26/3, 2022 at 13:14 Comment(3)
That's really scary. I don't like the fact that a corrupted Conda environment can cause hashlib to generate the spurious results.Coons
@Tan Did you by any chance find out what the reason for the corruption was? I just experienced the same, and environments breaking without warning is pretty scary indeed.Ribwort
@SaaruLindestøkke Sorry for the delayed answer. Unfortunately, I have not been able to pinpoint the exact problem...Tan
P
8

For ubuntu (jammy/focal).

Add this to your /etc/ssl/openssl.cnf to 're-enable' md4 to hashlib

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

Solution from this https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1971580/comments/3

Depending on your version or distribuition the path to the openssl.cnf file can be /usr/lib/openssl.cnf or other.

Pellitory answered 29/6, 2022 at 20:9 Comment(4)
Worked like a charm (Y). You saved the day, bro! The only difference I had is to add these line in /etc/ssl/openssl.cnfPulverulent
/usr/lib actually looks like a typo, shouldn't it be /etc/lib ?Piderit
@antonone thanks I think that depends on the distro, version ubuntu too.Pellitory
Also works on Rocky 9 (redhat) 9, thank'sFinsen
N
8

Update as of May 2023

This has been addressed in requests-ntlm>=1.2.0 which now uses SPNEGO: https://github.com/requests/requests-ntlm/pull/126

OpenSSL configuration changes should no longer be required with this version.

Tip: Since this is one of the top results for "unsupported hash type md4" (an error I encountered when using Ansible to target Windows on a modern Linux distro) simply updating this library in your current Ansible install appears to be a solution:

python3 -m pip install -U requests-ntlm
Neb answered 26/5, 2023 at 19:15 Comment(0)
T
2

Well, it seems that there was something corrupted in my conda environment. I created a new identical one, and it's been working ever since without having to change anything else.

Tan answered 26/3, 2022 at 13:14 Comment(3)
That's really scary. I don't like the fact that a corrupted Conda environment can cause hashlib to generate the spurious results.Coons
@Tan Did you by any chance find out what the reason for the corruption was? I just experienced the same, and environments breaking without warning is pretty scary indeed.Ribwort
@SaaruLindestøkke Sorry for the delayed answer. Unfortunately, I have not been able to pinpoint the exact problem...Tan
M
1

In my case, I was running a docker image python:3.7 and didn't specify the sub-version. The fix mentioned by @imbr did work. I found the openssl.cnf file location by running openssl version -d

But also, on my dev station, it was using python:3.7.15 and on the live server was python:3.7.17 From the release notes, you can see that openssl got an update on 17, https://www.python.org/downloads/release/python-3717/

Mamey answered 20/6, 2023 at 23:1 Comment(0)
P
-1
hashlib.new('md4', "test".encode()).hexdigest()
Phenanthrene answered 17/7, 2022 at 17:1 Comment(1)
Remember that Stack Overflow isn't just intended to solve the immediate problem, but also to help future readers find solutions to similar problems, which requires understanding the underlying code. This is especially important for members of our community who are beginners, and not familiar with the syntax. Given that, can you edit your answer to include an explanation of what you're doing and why you believe it is the best approach?Weak

© 2022 - 2024 — McMap. All rights reserved.