Minio Python Client: Upload Bytes directly

Question

I read the minio docs and I see two methods to upload data:

• put_object() this needs a io-stream
• fput_object() this reads a file on disk

I want to test minio and upload some data I just created with numpy.random.bytes().

How to upload data which is stored in a variable in the python interpreter?

Thirddegree

Answer 1

Take a look at io.BytesIO. These allow you to wrap byte arrays up in a stream which you can give to minio.

For example:

import io
from minio import Minio

value = "Some text I want to upload"
value_as_bytes = value.encode('utf-8')
value_as_a_stream = io.BytesIO(value_as_bytes)

client = Minio("my-url-here", ...) # Edit this bit to connect to your Minio server
client.put_object("my_bucket", "my_key", value_as_a_stream , length=len(value_as_bytes))

Pennsylvanian

Answer 2

I was in a similar situation: trying to store a pandas DataFrame as a feather file into minio. I needed to store bytes directly using the Minio client. In the end the code looked like that:

from io import BytesIO
from pandas import df
from numpy import random
import minio

# Create the client
client = minio.Minio(
    endpoint="localhost:9000",
    access_key="access_key",
    secret_key="secret_key",
    secure=False
)

# Create sample dataset
df = pd.DataFrame({
    "a": numpy.random.random(size=1000),
})

# Create a BytesIO instance that will behave like a file opended in binary mode 
feather_output = BytesIO()
# Write feather file
df.to_feather(feather_output)
# Get numver of bytes
nb_bytes = feather_output.tell()
# Go back to the start of the opened file
feather_output.seek(0)

# Put the object into minio
client.put_object(
    bucket_name="datasets",
    object_name="demo.feather", 
    length=nb_bytes,
    data=feather_output
)

I had to use .seek(0) in order for minio to be able to insert correct amounts of bytes.

Expeditious

what if i want to save it as csv file. – Consequential

Simply replace BytesIO() by StringIO() and use to_csv() instead of to_feather(). This should do the trick – Expeditious

Answer 3

@gcharbon: this solution does not work for me. client.put_object() does only accept bytes like objects.

Here is my solution:

from minio import Minio 
import pandas as pd
import io  

#Can use a string with csv data here as well
csv_bytes = df.to_csv().encode('utf-8')

csv_buffer = io.BytesIO(csv_bytes)

# Create the client
client = Minio(
    endpoint="localhost:9000",
    access_key="access_key",
    secret_key="secret_key",
    secure=False
)

client.put_object("bucketname", 
                  "objectname",  
                  data=csv_buffer, 
                  length=len(csv_bytes), 
                  content_type='application/csv')

Collagen