Paperclip + S3: Migrating existing files from one :path format to another

Question

I have a model with a avatar paperclip attach. It has now a plain standard path

has_attached_file :avatar,
  :path => "/:id-:style-:filename"

Which I need to move into an obfuscated one

has_attached_file :avatar,
  :path => "/:id-:hash.:extension"
  :hash_secret => 'asecuresecret'

Everything's working fine with new images, but how do I migrate the existing files stored in S3 to the new path format, ie rename them?

I'm using paperclip 2.7 and Rails 3.2

Thanks !!!

Thicken

For those interested, moving from hash to non-hash is a little more interesting and the best method I've found is to create a new has_attached_file to make the migration. – Derisive

Answer 1

This rake task should do the trick. I tried it with aws-sdk 1.5.2 and ruby 1.9.3p194.

The new_key should map to your new paperclip path. Don't forget to set :acl according to your needs.

namespace :data do
  desc 'aws images migration'
  task :migrate_images do |t, args|
    s3 = AWS::S3.new(:access_key_id => 'XXX', :secret_access_key => 'XXX')
    bucket = s3.buckets['your-bucket-name']
    bucket.objects.each do |object|
      new_key = object.key.gsub(........)
      new_object = bucket.objects[new_key]
      object.copy_to new_object, {:acl => :public_read}
    end
  end
end

The original file should be deleted manually or using a similar task, once you are sure the new file is correct.

Trier

Tested with aws-sdk 1.26 and ruby 2.0. Thanks and sorry for the delay in the acceptance! – Thicken

Answer 2

If you want to work only with Paperclip and you are not worried about re-uploading I followed another approach.

Let's suppose you have the following:

class User
   has_attached_file :image, path: "/:old_path/:filename"
   ...
end

and you want to migrate to the new path: "/:new_path/:filename"

my suggestion is to create a FakeUser with the old path and change it in the User model.

class FakeUser
   self.table_name = :users
   has_attached_file :image, path: "/:old_path/:filename"
   ...
end

class User
   has_attached_file :image, path: "/:new_path/:filename"
   ...
end

You can now write the following migration:

FakeUser.find_each do |fake_user|
   User.find(fake_user.id).update(image: fake_user.image)
   fake_user.image.destroy
end

You can then delete the FakeUser model when the migration is finished.

By the way, this approach will work perfectly also to migrate from local filesystem to S3 or vice-versa.

Undaunted

this did work for me, I even could simplify it by inheriting the fake model from the real model class FakeUser < User – Darn

Very good solution to migrate all your files! Better than the scripts that I have saw. – Overzealous

Answer 3

I would write a rake task (or just a plain script if you prefer, to be run in the rails context). If you're using the aws-s3 gem, iterate over the instances of the model which you know they have the old path format or try writing some condition on the filename to match them, and the move one by one.

Model.find_in_batches(:batch_size => 500,
      :conditions => "avatar_filename like 'SOMETHING_MATCHING'") do |o|
  AWS::S3::S3Object.rename(old_path(o.avatar), o.avatar.url, 'BUCKET_NAME')
end

If you have already configured avatar with the new path definition, write a method that can build the old path based on the avatar properties.

You can read the aws-s3 gem docs here to see how to establish a connection to your S3 account.

Panthia