Python - HTTP module cannot parse response if the server answers before the PUT is complete

Question

I'm using the requests (which uses urllib3 and the Python http module under the hood) library to upload a file from a Python script. My backend starts by inspecting the headers of the request and if it doesn't comply with the needed prerequisites, it stops the request right away and respond with a valid 400 response.

This behavior works fine in Postman, or with Curl; i.e. the client is able to parse the 400 response even though it hasn't completed the upload and the server answers prematurely. However, while doing so in Python with requests/urllib3, the library is unable to process the backend response :

Traceback (most recent call last):
  File "C:\Users\Neumann\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\Neumann\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\urllib3\connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1049, in _send_output
    self.send(chunk)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 971, in send
    self.sock.sendall(data)
ConnectionResetError: [WinError 10054] Une connexion existante a dû être fermée par l’hôte distant

Because the server answers before the transfer is complete, it mistakenly considers that the connection has been aborted, even though the server DOES return a valid response.

Is there a way to avoid this and parse the response nonetheless ?

Steps to reproduce the issue :

• Download minIO : https://min.io/download#/
• Run minIO :

export MINIO_ACCESS_KEY=<access_key>
export MINIO_SECRET_KEY=<secret_key>
.\minio.exe server <data folder>

• Run the following script :

import os
import sys
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

def fatal(msg):
    print(msg)
    sys.exit(1)

def upload_file():
    mp_encoder = MultipartEncoder(fields={'file': (open('E:/Downloads/kek.mp3', 'rb'))})
    headers = { "Authorization": "invalid" }

    print('Uploading file with headers : ' + str(headers))

    upload_endpoint = 'http://localhost:9000/mybucket/myobject'
    try:
        r = requests.put(upload_endpoint, headers=headers, data=mp_encoder, verify=False)
    except requests.exceptions.ConnectionError as e:
        print(e.status)
        for property, value in vars(e).items():
            print(property, ":", value)
        fatal(str(e))
    
    if r.status_code != 201:
        for property, value in vars(r).items():
            print(property, ":", value)
        fatal('Error while uploading file. Status ' + str(r.status_code))
    print('Upload successfully completed')

if __name__ == "__main__":
    upload_file()

If you change the request line with this, it will work (i.e. the server returns 400 and the client is able to parse it) :

r = requests.put(upload_endpoint, headers=headers, data='a string', verify=False)

EDIT : I updated the traceback and changed the question title to reflect the fact that it's neither requests or urllib3 fault, but the Python http module that is used by both of them.

Plagiarism

Can you add a full example for that? Is there a public site that have this happening with this? – Frowsy

Can you also share the relevant code from the backend? Which web server do you use and how is it configured? Also, please share the working curl request. – Flake

I edited the question to include reproduction steps and complete script. – Plagiarism

What is MultipartEncoder? Can you try as it is documented at Streaming Uploads? with open('E:/Downloads/kek.mp3', 'rb') as f: requests.put('http://localhost:9000/mybucket/myobject', data=f) – Bleareyed

This question is related: #14251491 – Estienne

This is a recorded problem, and there seems to be no way to fix this with the http module. I cannot find an alternate http module; maybe a server-oriented module like flask could somehow be used to handle this. – Gine

Which version of urllib3 are you using? – Metalliferous

Answer 1

This problem should be fixed in urllib3 v1.26.0. What version are you running?

The problem is that the server closes the connection after it responds with 400, so the socket is closed when urllib3 tries to keep sending data to it. So it isn't really mistakenly thinking that the connection is closed, it just mishandles that situation.

Your example code works fine on my machine with urllib3==1.26.0 . But I notice that you get a different exception on your Windows machine, so it might be that the fix doesn't work. In that case, I would just catch the exception and file a bug report to the maintainers of urllib3.

Metalliferous

Indeed, after upgrading to the latest versions of requests/urllib3, it does work, but only on Linux, not on Windows. Luckily my script will be mainly used on Linux platforms, so that's good enough for me. Thanks. – Plagiarism

Answer 2

you should try requests.get instead of put and thans check if it works . Please find my sample code below .

try:
    output = requests.get('http://' + <ipaddress>)
    status = output.status_code
    print(status)
    if status == 200:
        print("PASS: HTTP is successful")
    else:
        raise RuntimeError("FAIL: HTTP is not successful")
except:
    Flag = Flag + 1
    print("FAIL: HTTP is not successful ")

this code of mine works fine with python3

Moniquemonism

Having another API to "dry-run" the parameters before using the actual upload API is not an option that I have – Plagiarism