Write in memory object to S3 via boto3

Question

I am attempting to write files directly to S3 without creating a local file which is then uploaded.

I am using cStringIO to generate a file in memory, but I am having trouble figuring out the proper way to upload it in boto3.

def writetos3(sourcedata, filename, folderpath):
     s3 = boto3.resource('s3')
     data = open(sourcedata, 'rb')
     s3.Bucket('bucketname').put_object(Key= folderpath + "/" + filename, Body=data)

Above is the standard boto3 method that I was using previously with the local file, it does not work without a local file, I get the following error: coercing to Unicode: need string or buffer, cStringIO.StringO found .

Because the in memory file (I believe) is already considered open, I tried changing it to the code below, but it still does not work, no error is given the script simply hangs on the last line of the method.

def writetos3(sourcedata, filename, folderpath):
    s3 = boto3.resource('s3')
    s3.Bucket('bucketname').put_object(Key= folderpath + "/" + filename, Body=sourcedata)

Just for more info, the value I am attempting to write looks like this

(cStringIO.StringO object at 0x045DC540)

Does anyone have an idea of what I am doing wrong here?

Stealing

Do you get an error? – Donative

When I use the original method the error I get is this: TypeError: coercing to Unicode: need string or buffer, cStringIO.StringO found. When I use the second method the program just hangs, getting stuck on the last line of the writetos3 method – Stealing

Answer 1

It looks like you want this:

    data = open(sourcedata, 'rb').decode()

It defaults to utf8. Also, I encourage you to run your code under python3, and to use appropriate language tags for your question.

Typal

Thanks for the suggestions, I was planning on upgrading to 3.0 but I just went ahead and converted the script now. After converting the syntax and running the method with your suggested change I am now getting the following error: TypeError: expected str, bytes or os.PathLike object, not _io.StringIO – Stealing

I ended up solving the problem by simply converting the stringIO object to a string before passing it to the writetoS3 method. – Stealing