How to upload file with python requests?

Question

I'm performing a simple task of uploading a file using Python requests library. I searched Stack Overflow and no one seemed to have the same problem, namely, that the file is not received by the server:

import requests
url='http://nesssi.cacr.caltech.edu/cgi-bin/getmulticonedb_release2.cgi/post'
files={'files': open('file.txt','rb')}
values={'upload_file' : 'file.txt' , 'DB':'photcat' , 'OUT':'csv' , 'SHORT':'short'}
r=requests.post(url,files=files,data=values)

I'm filling the value of 'upload_file' keyword with my filename, because if I leave it blank, it says

Error - You must select a file to upload!

And now I get

File  file.txt  of size    bytes is  uploaded successfully!
Query service results:  There were 0 lines.

Which comes up only if the file is empty. So I'm stuck as to how to send my file successfully. I know that the file works because if I go to this website and manually fill in the form it returns a nice list of matched objects, which is what I'm after. I'd really appreciate all hints.

Some other threads related (but not answering my problem):

• Send file using POST from a Python script
• http://docs.python-requests.org/en/latest/user/quickstart/#response-content
• Uploading files using requests and send extra data
• http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow

Larousse

Answer 1

If upload_file is meant to be the file, use:

files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}

r = requests.post(url, files=files, data=values)

and requests will send a multi-part form POST body with the upload_file field set to the contents of the file.txt file.

The filename will be included in the mime header for the specific field:

>>> import requests
>>> open('file.txt', 'wb')  # create an empty demo file
<_io.BufferedWriter name='file.txt'>
>>> files = {'upload_file': open('file.txt', 'rb')}
>>> print(requests.Request('POST', 'http://example.com', files=files).prepare().body.decode('ascii'))
--c226ce13d09842658ffbd31e0563c6bd
Content-Disposition: form-data; name="upload_file"; filename="file.txt"


--c226ce13d09842658ffbd31e0563c6bd--

Note the filename="file.txt" parameter.

You can use a tuple for the files mapping value, with between 2 and 4 elements, if you need more control. The first element is the filename, followed by the contents, and an optional content-type header value and an optional mapping of additional headers:

files = {'upload_file': ('foobar.txt', open('file.txt','rb'), 'text/x-spam')}

This sets an alternative filename and content type, leaving out the optional headers.

If you are meaning the whole POST body to be taken from a file (with no other fields specified), then don't use the files parameter, just post the file directly as data. You then may want to set a Content-Type header too, as none will be set otherwise. See Python requests - POST data from a file.

Blockbusting

Hi, How do I send multiple files sharing a same name? Like 'attachment' for example. – Revell

@William: you can use a sequence of 2-value tuples too, which lets you re-use field names: files = [('attachment', open('attachment1.txt', 'rb')), ('attachment', open('attachment2.txt', 'rb'))]. Each tuple is a pair of key and value. – Blockbusting

Also you can also use files={'file':('nameoffile',open('namoffile','rb'),'Content-Type':'text/html','other header'),'file2':('nameoffile2',open('nameoffile2','rb'),'Content-Type':'application/xml','other header')} but If files={} is used then headers={'Content-Type':'blah blah'} must not be used! -> @martijn-pieters: bacause the multipart/form-data Content-Type must include the boundary value used to deliniate the parts in the post body. Not setting the Content-Type header ensures that requests sets it to the correct value. – Coronagraph

@zaki: don't conflate the mime type of individual parts with the content type of the POST request. Your syntax in also incorrect, be careful when mixing tuples and dictionaries, and if you are only setting the mime type, just use a string in the 3rd position of a 3-element tuple: files={'file':('nameoffile', open('nameoffile', 'rb'), text/html'), 'file2': ('nameoffile2',open('nameoffile2', 'rb'), 'application/xml')}. – Blockbusting

All this is covered by the requests quickstart and advanced documentation. – Blockbusting

I'm getting corrupted files. It is adding strange text like 'Content-Disposition: form-data; name="upload_file"; filename="file.txt"' onto the end of my files. Any idea why? – Guest

@johnktejik: then you did something wrong with the boundary. You are probably setting the Content-Type header manually and therefor clobbering the boundary parameter that must be present in that header for multipart/form-data to work. When using files do not set the Content-Type header, leave that to the library to do. The 'extra data' you see is mime data for each part, see How does HTTP file upload work? – Blockbusting

This will get error: NotImplementedError: Streamed bodies and files are mutually exclusive. – Tadtada

@secsilm: not with the answer posted here. That error requires that you also have set data to something that's iterable (other than a string, tuple, list or dictionary). Did you accidentally pass in something other than a dict as shown in this answer? – Blockbusting

Answer 2

(2018) the new python requests library has simplified this process, we can use the 'files' variable to signal that we want to upload a multipart-encoded file

url = 'http://httpbin.org/post'
files = {'file': open('report.xls', 'rb')}

r = requests.post(url, files=files)
r.text

Threaten

Does requests library automatically close the file? – Marcy

hello, its been awhile since I've used this library. nice question. could you give me and the others a hand by typing lsof | grep "filename" and share your results with us? thanks :) – Threaten

With the use of lsof, is seems that the file remains open, or at least, this is how I interpret the following results. Before, running the open there is no record in lsof table about the filename. Then after the open is executed, multiple records appear with read access. After executing the requests.post, the records are still there indicating that the file did not close. – Marcy

(2021) If you also need parameters when upload a file, you can add params like this: r = requests.post(url,files=files,params={"key":value}) – Religiose

What "new python requests library"? This 2018 solution isn't any different from the 2014 solution. – Allophone

Answer 3

Client Upload

If you want to upload a single file with Python requests library, then requests lib supports streaming uploads, which allow you to send large files or streams without reading into memory.

with open('massive-body', 'rb') as f:
    requests.post('http://some.url/streamed', data=f)

Server Side

Then store the file on the server.py side such that save the stream into file without loading into the memory. Following is an example with using Flask file uploads.

@app.route("/upload", methods=['POST'])
def upload_file():
    from werkzeug.datastructures import FileStorage
    FileStorage(request.stream).save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
    return 'OK', 200

Or use werkzeug Form Data Parsing as mentioned in a fix for the issue of "large file uploads eating up memory" in order to avoid using memory inefficiently on large files upload (s.t. 22 GiB file in ~60 seconds. Memory usage is constant at about 13 MiB.).

@app.route("/upload", methods=['POST'])
def upload_file():
    def custom_stream_factory(total_content_length, filename, content_type, content_length=None):
        import tempfile
        tmpfile = tempfile.NamedTemporaryFile('wb+', prefix='flaskapp', suffix='.nc')
        app.logger.info("start receiving file ... filename => " + str(tmpfile.name))
        return tmpfile

    import werkzeug, flask
    stream, form, files = werkzeug.formparser.parse_form_data(flask.request.environ, stream_factory=custom_stream_factory)
    for fil in files.values():
        app.logger.info(" ".join(["saved form name", fil.name, "submitted as", fil.filename, "to temporary file", fil.stream.name]))
        # Do whatever with stored file at `fil.stream.name`
    return 'OK', 200

Herzog

Thanks for this answer! I'm looking a bit more into how to upload multiple files using the streaming upload, but most examples are re-using the one you shared with a single open(). Would you know how to do that? – Powerboat

This doesn't create a "multipart/form-data" post request. – Allophone

Answer 4

You can send any file via post api while calling the API just need to mention files={'any_key': fobj}

import requests
import json
    
url = "https://request-url.com"
 
headers = {"Content-Type": "application/json; charset=utf-8"}
    
with open(filepath, 'rb') as fobj:
    response = requests.post(url, headers=headers, files={'file': fobj})
 
print("Status Code", response.status_code)
print("JSON Response ", response.json())

Zeebrugge

Answer 5

@martijn-pieters answer is correct, however I wanted to add a bit of context to data= and also to the other side, in the Flask server, in the case where you are trying to upload files and a JSON.

From the request side, this works as Martijn describes:

files = {'upload_file': open('file.txt','rb')}
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}

r = requests.post(url, files=files, data=values)

However, on the Flask side (the receiving webserver on the other side of this POST), I had to use form

@app.route("/sftp-upload", methods=["POST"])
def upload_file():
    if request.method == "POST":
        # the mimetype here isnt application/json
        # see here: https://mcmap.net/q/73949/-how-to-get-posted-json-in-flask
        body = request.form
        print(body)  # <- immutable dict

body = request.get_json() will return nothing. body = request.get_data() will return a blob containing lots of things like the filename etc.

Here's the bad part: on the client side, changing data={} to json={} results in this server not being able to read the KV pairs! As in, this will result in a {} body above:

r = requests.post(url, files=files, json=values). # No!

This is bad because the server does not have control over how the user formats the request; and json= is going to be the habbit of requests users.

Wrangle

@martijn-pieters just seeing if youd like to include any of this in your answer, and I will delete this; this might be useful for people coming to this answer from a "both sides" of the client and server perspective. – Wrangle

hell yeah it was useful ! – Firkin

Answer 6

Upload:

with open('file.txt', 'rb') as f:
    files = {'upload_file': f.read()}
    
values = {'DB': 'photcat', 'OUT': 'csv', 'SHORT': 'short'}

r = requests.post(url, files=files, data=values)

---

Download (Django):

with open('file.txt', 'wb') as f:
    f.write(request.FILES['upload_file'].file.read())

Blasien

Answer 7

Regarding the answers given so far, there was always something missing that prevented it to work on my side (although all the hints in the answers were helpful). So let me summarize from my own experiments what worked for me finally:

import json
import os
import requests

API_ENDPOINT = "http://localhost:80"
access_token = "sdfJHKsdfjJKHKJsdfJKHJKysdfJKHsdfJKHs"  # TODO: get fresh Token here


def upload_engagement_file(filepath):
 
    url = API_ENDPOINT + "/api/files"  # add any URL parameters if needed
    hdr = {"Authorization": "Bearer %s" % access_token}
    with open(filepath, "rb") as fobj:
        file_obj = fobj.read()
        file_basename = os.path.basename(filepath)
        file_to_upload = {"file": (str(file_basename), file_obj)}
        finfo = {"fullPath": filepath}
        upload_response = requests.post(url, headers=hdr, files=file_to_upload, data=finfo)
        fobj.close()
    # print("Status Code ", upload_response.status_code)
    # print("JSON Response ", upload_response.json())
    return upload_response

Note that requests.post(...) needs

• a url parameter, containing the full URL of the API endpoint you're calling, using the API_ENDPOINT, assuming we have an http://localhost:8000/api/files endpoint to POST a file
• a headers parameter, containing at least the authorization (bearer token)
• a files parameter taking the name of the file plus the entire file content
• a data parameter taking just the path and file name

Installation required (console):

pip install requests

What you get back from the function call is a response object containing a status code and also the full error message in JSON format. The commented print statements at the end of upload_engagement_file are showing you how you can access them.

Note: Some useful additional information about the requests library can be found here

Rawls

Answer 8

Some may need to upload via a put request and this is slightly different that posting data. It is important to understand how the server expects the data in order to form a valid request. A frequent source of confusion is sending multipart-form data when it isn't accepted. This example uses basic auth and updates an image via a put request.

url = 'foobar.com/api/image-1'
basic = requests.auth.HTTPBasicAuth('someuser', 'password123')
# Setting the appropriate header is important and will vary based
# on what you upload
headers = {'Content-Type': 'image/png'} 
with open('image-1.png', 'rb') as img_1:
    r = requests.put(url, auth=basic, data=img_1, headers=headers)

While the requests library makes working with http requests a lot easier, some of its magic and convenience obscures just how to craft more nuanced requests.

Selfhelp

Answer 9

In Ubuntu you can apply this way,

to save file at some location (temporary) and then open and send it to API

      path = default_storage.save('static/tmp/' + f1.name, ContentFile(f1.read()))
      path12 = os.path.join(os.getcwd(), "static/tmp/" + f1.name)
      data={} #can be anything u want to pass along with File
      file1 = open(path12, 'rb')
      header = {"Content-Disposition": "attachment; filename=" + f1.name, "Authorization": "JWT " + token}
       res= requests.post(url,data,header)

Jidda

what is the value of data variable? – Discernible

it can be anything like user name,I've just shown how to upload files to REST apis – Jidda

I see no reason to make a temporary file. And you aren't even using file1 anywhere. Also Content-Disposition doesn't belong there. – Allophone