Ruby - Append content at the end of the existing s3 file using fog

Question

How to append text in an existing or newly created file in S3. I am using fog and I have following code

require 'fog'
file    = "abc.csv"
bucket  = 'my_bucket'
storage = Fog::Storage.new(:provider => 'AWS', :aws_access_key_id => 'XXXXXXXX', :aws_secret_access_key => 'YYYYYYYY')
dir     = connection.directories.new(:key => bucket) # no harm, if this bucket already exists, if not create one
buffer  = ["big_chunk1", "big_chunk2", "big_chunk3", "big_chunk4", "big_chunk5"]

# I need help after this line. No changes above.
buffer.each do |chunk|
  # this will not work as it will not append text below the existing file or 
  # newly created one. I am not looking for solution suggesting, buffer.join('') 
  # and then write whole chunk at once. I have to write in chuck for some specific reason.
  # Also I dont want to first read existing data and then take in to memory
  # and then append and finally write back.
  dir.files.create(:key => file, :body => chunk, :public => false) 
end

Frasco

Answer 1

Once uploaded to S3, a file is immutable - it cannot be changed or appended to.

If you a just looking to upload a file in chunks, you may be able to use the multipart upload api, (assuming you have under 10000 chunks and that all but the last will be at least 5MB), however you'd probably need to drop to the fog request level (instead of using the fog models as you are currently).

Baggett

Thanks @Frederick Cheung. I have reached to the same conclusion after having so much research. By the way, do you know how to sign an s3 url using fog? Could not find any info about it easily in fog documentation. – Frasco