How do I replicate a `curl` with `-F` in Python 3?

Question

I have this curl:

curl -v "http://some.url" \
    -X PUT \
    -H "X-API-Key: API_KEY" \
    -H 'Accept: application/json' \
    -H 'Content-Type: multipart/form-data' \
    -F "logo=@the_logo.png;type=image/png"

And I am trying to replicate this with Python. What I have tried so far is:

import requests


with open("the_logo.png", "rb") as file_obj:
   data = file_obj.read()
requests.put(
    "http://some.url",
    files={"logo": ("the_logo.png", data, "image/png")},
    headers={
        "X-API-Key": "API_KEY",
        "Accept": "application/json",
        "Content-Type": "multipart/form-data"}

But for some reason the curl above works, while the Python code does not and the server replies with a 422.

How can I make Python replicate what curl does?

Grief

Have you sniffed the HTTP requests to see how they differ? – Deucalion

BTW, you should be able to have file_obj itself be in the data location in the tuple (if you haven't read from it and the pointer is still at the front of the file, so you'd need to take out the data = file_obj.read() line). – Deucalion

try files={"logo": ("the_logo.png", open('the_logo.png', 'rb'), "image/png")} once. – Pontificals

have you checked out pycurl.io/docs/latest/index.html? – Homophonous

@Tarique's suggestion matches with this answer, except using POST instead of PUT: How to upload file with python requests? – Hermetic

Tarique's comment is what I see from curl.trillworks.com/#python – Diena

@CharlesDuffy How would I go with that? It seems that Python/requests does not provide the raw data sent – Grief

@Pontificals I tried that but I get the same problem – Grief

The word "sniff" implies use of a packet sniffer -- Wireshark, tcpdump, etc. – Deucalion

Answer 1

After some reading, it appears that the trick is to NOT set Content-Type in the headers when using requests and the files parameter.

Grief

Answer 2

So, curl -F ... is basically emulating submitting a form.

-F, --form <name=content>

(HTTP SMTP IMAP) For HTTP protocol family, this lets curl emulate a filled-in form in which a user has pressed the submit button. This causes curl to POST data using the Content-Type multipart/form-data according to RFC 2388.

You can replicate this in Python using the requests module (note that you will need to install it - e.g., pip3 install requests):

import requests

with open("the_logo.png", "rb") as file:
    file_content = file.read()

response = requests.put(
    "http://localhost:8080",
    files={"logo": file_content},
    headers={
        "X-API-Key": "API_KEY",
        "Accept": "application/json"
    }
)

Or, although not really straightforward, using the built-in urllib.request module:

import io
from urllib.request import urlopen, Request
import uuid

with open("the_logo.png", "rb") as file:
    file_content = file.read()

boundary = uuid.uuid4().hex.encode("utf-8")

buffer = io.BytesIO()
buffer.write(b"--" + boundary + b"\r\n")
buffer.write(b"Content-Disposition: form-data; name=\"logo\"; filename=\"logo\"\r\n")
buffer.write(b"\r\n")
buffer.write(file_content)
buffer.write(b"\r\n")
buffer.write(b"--" + boundary + b"--\r\n")
data = buffer.getvalue()

request = Request(url="http://localhost:8080", data=data, method="PUT")
request.add_header("X-API-Key", "API_KEY")
request.add_header("Accept", "application/json")
request.add_header("Content-Length", len(data))
request.add_header("Content-Type", f"multipart/form-data; boundary={boundary.decode('utf-8')}")
with urlopen(request) as response:
    response_body = response.read().decode("utf-8")

If you use for example netcat to listen to the used port, you should see that both curl -F ... and the above scripts generate something like the following:

$ nc -l -p 8080
PUT / HTTP/1.1
Host: localhost:8080
X-Api-Key: API_KEY
Accept: application/json
Content-Length: [DATA_LENGTH]
Content-Type: multipart/form-data; boundary=[GENERATED_BOUNDARY]
Connection: close

--[GENERATED_BOUNDARY]
Content-Disposition: form-data; name="logo"; filename="logo"

[PNG_BYTES]
--[GENERATED_BOUNDARY]--

NOTE: For a more complete urllib.request implementation see https://pymotw.com/3/urllib.request/#uploading-files

Johst