Uploading multiple files in a single request using python requests module

Question

The Python requests module provides good documentation on how to upload a single file in a single request:

 files = {'file': open('report.xls', 'rb')}

I tried extending that example by using this code in an attempt to upload multiple files:

 files = {'file': [open('report.xls', 'rb'), open('report2.xls, 'rb')]}

but it resulted in this error:

 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py",      line 1052, in splittype
 match = _typeprog.match(url)
 TypeError: expected string or buffer

Is it possible to upload a list of files in a single request using this module, and how?

Coopt

why has there not been an accepted answer? Does the high vote answer below not suffice? – Generate

Ping/bumping. Do any of these answers suffice? – Gage

Answer 1

To upload a list of files with the same key value in a single request, you can create a list of tuples with the first item in each tuple as the key value and the file object as the second:

files = [('file', open('report.xls', 'rb')), ('file', open('report2.xls', 'rb'))]

Sobersided

Lets say I have a list of file names, is there a way to do this with a list comprehension? – Damascene

@MatthewSemik file_names = ["abc.txt", "pqr.txt" ] files = [('file', open(f, 'rb')) for f in file_names] – Giacomo

will it give an error if you send this list through request post? – Terrorism

Answer 2

Multiple files with different key values can be uploaded by adding multiple dictionary entries:

files = {'file1': open('report.xls', 'rb'), 'file2': open('otherthing.txt', 'rb')}
r = requests.post('http://httpbin.org/post', files=files)

Ivie

Interesting. Will try your approach. The reason I tried list was because Flask (python web framework) says files is a multidict and the way to access all the files upload would be to do: request.files.getall('file') – Coopt

Do I need to close the file descriptors by myself? or will they be automatically closed like with open('file', 'r') as f..? – Idempotent

@Lukasa, is there a solution similar approach in R ? – Hitchhike

The limitation of using a dictionary rather than a list of tuples is the fact that it doesn't allow you to specify multiple files under the same keyword, which is a completely legitimate and not rare use case of multipart requests – Macmahon

Answer 3

The documentation contains a clear answer.

Quoted:

You can send multiple files in one request. For example, suppose you want to upload image files to an HTML form with a multiple file field ‘images’:

To do that, just set files to a list of tuples of (form_field_name, file_info):

url = 'http://httpbin.org/post'
multiple_files = [('images', ('foo.png', open('foo.png', 'rb'), 'image/png')),
                      ('images', ('bar.png', open('bar.png', 'rb'), 'image/png'))]
r = requests.post(url, files=multiple_files)
r.text

# {
#  ...
#  'files': {'images': 'data:image/png;base64,iVBORw ....'}
#  'Content-Type': 'multipart/form-data; boundary=3131623adb2043caaeb5538cc7aa0b3a',
#  ...
# }

Generate

What form can file_info take? Can I omit the content-type? What else can be part of file_info? the documentation doesn't go into detail. – Rishi

@AmauryRodriguez I recommend you look at the source for all those details. – Generate

Answer 4

In case you have the files from a form and want to forward it to other URL or to API. Here is an example with multiple files and other form data to forward to other URL.

images = request.files.getlist('images')
files = []
for image in images:
    files.append(("images", (image.filename, image.read(), image.content_type)))
r = requests.post(url="http://example.com/post", data={"formdata1": "strvalue", "formdata2": "strvalue2"}, files=files)

Merlenemerlin

Exactly what I needed thank you so much!! – Nimbostratus

Answer 5

You need to create a file list to upload multiple images:

file_list = [  
       ('Key_here', ('file_name1.jpg', open('file_path1.jpg', 'rb'), 'image/png')),
       ('key_here', ('file_name2.jpg', open('file_path2.jpg', 'rb'), 'image/png'))
   ]

r = requests.post(url, files=file_list)

If you want to send files on the same key you need to keep the key same for each element, and for a different key just change the keys.

Source : https://stackabuse.com/the-python-requests-module/

Needful

Answer 6

I'm a little bit confused, but directly opening file in request (however same is written in official requests guide) is not so "safe".

Just try:

import os
import requests
file_path = "/home/user_folder/somefile.txt"
files = {'somefile': open(file_path, 'rb')}
r = requests.post('http://httpbin.org/post', files=files)

Yes, all will be ok, but:

os.rename(file_path, file_path)

And you will get:

PermissionError:The process cannot access the file because it is being used by another process

Please, correct me if I'm not right, but it seems that file is still opened and I do not know any way to close it.

Instead of this I use:

import os
import requests
#let it be folder with files to upload
folder = "/home/user_folder/"
#dict for files
upload_list = []
for files in os.listdir(folder):
    with open("{folder}{name}".format(folder=folder, name=files), "rb") as data:
        upload_list.append(files, data.read())
r = request.post("https://httpbin.org/post", files=upload_list)
#trying to rename uploaded files now
for files in os.listdir(folder):
    os.rename("{folder}{name}".format(folder=folder, name=files), "{folder}{name}".format(folder=folder, name=files))

Now we do not get errors, so I recommend to use this way to upload multiple files or you could receive some errors. Hope this answer well help somebody and save priceless time.

Diphenylhydantoin

I don't think what you're trying to do will work, because the file pointers will already be closed, before they're used by requests.post. Therefore, we're left with the option to have open pointers. We can, I think, put the pointers in variables, and after they're used, close them OR have parallel opens, post inside the with block. The pointers will automatically be closed. – Waterside

Answer 7

Using These methods File will Automatically be closed.

Method 1

with open("file_1.txt", "rb") as f1, open("file_2.txt", "rb") as f2:
    files = [f1, f2]
    response = requests.post('URL', files=files)

But When You're Opening Multiple Files This Can Get Pretty long

Method 2:

files = [open("forms.py", "rb"), open("data.db", "rb")]
response = requests.post('URL', files=files)

# Closing all Files
for file in files: 
    file.close()

Chorister

Answer 8

If you have multiple files in a python list, you can use eval() in a comprehension to loop over the files in the requests post files parameter.

file_list = ['001.jpg', '002.jpg', '003.jpg']
files=[eval(f'("inline", open("{file}", "rb"))') for file in file_list ]

requests.post(
        url=url,
        files=files
)

Cynewulf

Answer 9

In my case uploading all the images which are inside the folder just adding key with index

e.g. key = 'images' to e.g. 'images[0]' in the loop

 photosDir = 'allImages'
 def getFilesList(self):
        listOfDir = os.listdir(os.path.join(os.getcwd()+photosDir))
        setOfImg = []
        for key,row in enumerate(listOfDir):
            print(os.getcwd()+photosDir+str(row) , 'Image Path')
            setOfImg.append((
                'images['+str(key)+']',(row,open(os.path.join(os.getcwd()+photosDir+'/'+str(row)),'rb'),'image/jpg')
            ))
        print(setOfImg)
        return  setOfImg

Breadstuff

Answer 10

The way of files = [("file", (filename, fileobj)), ("file", (filename, fileobj))] in the other answers weren't working for me, but files={"file1": (filename, fileobj), "file2": (filename, fileobj)} did. Just an alternative method in case the above answers don't work

Ripon

Answer 11

To upload multiple files in single request,

files = [("files", (file.filename, file.file.read(), file.content_type)) 
            for file in files]
requests.post(url=url, 
              data=data, 
              headers=headers, 
              files=files)

Diazo