Rails 4 and paperclip - Stop the :original style file upload to copy it from an S3 remote directory

Question

I use Paperclip 4.0.2 and in my app to upload pictures.

So my Document model has an attached_file called attachment

The attachment has few styles, say :medium, :thumb, :facebook

In my model, I stop the styles processing, and I extracted it inside a background job.

class Document < ActiveRecord::Base
# stop paperclip styles generation    
before_post_process
  false
end

But the :original style file is still uploaded!

I would like to know if it's possible to stop this behavior and copy the file inside the :original/filename.jpg from a remote directory My goal being to use a file that has been uploaded in a S3 /temp/ directory with jQuery File upload, and copy it to the directory where Paperclip needs it to generate the others styles.

Thank you in advance for your help!

Pullet

Answer 1

New Answer:

paperclip attachments get uploaded in the flush_writes method which, for your purposes, is part of the Paperclip::Storage::S3 module. The line which is responsible for the uploading is:

s3_object(style).write(file, write_options)

So, by means of monkey_patch, you can change this to something like:

s3_object(style).write(file, write_options) unless style.to_s == "original" and @queued_for_write[:your_processed_style].present?

EDIT: this would be accomplished by creating the following file: config/initializers/decorators/paperclip.rb

Paperclip::Storage::S3.class_eval do
  def flush_writes #:nodoc:
    @queued_for_write.each do |style, file|
    retries = 0
      begin
        log("saving #{path(style)}")
        acl = @s3_permissions[style] || @s3_permissions[:default]
        acl = acl.call(self, style) if acl.respond_to?(:call)
        write_options = {
          :content_type => file.content_type,
          :acl => acl
        }

        # add storage class for this style if defined
        storage_class = s3_storage_class(style)
        write_options.merge!(:storage_class => storage_class) if storage_class

        if @s3_server_side_encryption
          write_options[:server_side_encryption] = @s3_server_side_encryption
        end

        style_specific_options = styles[style]

        if style_specific_options
          merge_s3_headers( style_specific_options[:s3_headers], @s3_headers, @s3_metadata) if style_specific_options[:s3_headers]
          @s3_metadata.merge!(style_specific_options[:s3_metadata]) if style_specific_options[:s3_metadata]
        end

        write_options[:metadata] = @s3_metadata unless @s3_metadata.empty?
        write_options.merge!(@s3_headers)

        s3_object(style).write(file, write_options) unless style.to_s == "original" and @queued_for_write[:your_processed_style].present?
      rescue AWS::S3::Errors::NoSuchBucket
        create_bucket
        retry
      rescue AWS::S3::Errors::SlowDown
        retries += 1
        if retries <= 5
          sleep((2 ** retries) * 0.5)
          retry
        else
          raise
        end
      ensure
        file.rewind
      end
    end
    after_flush_writes # allows attachment to clean up temp files
    @queued_for_write = {}
  end
end

now the original does not get uploaded. You could then add some lines, like those of my origninal answer below, to your model if you wish to transfer the original to its appropriate final location if it was uploaded to s3 directly.

Original Answer:

perhaps something like this placed in your model executed with the after_create callback:

    paperclip_file_path = "relative/final/destination/file.jpg"
    s3.buckets[BUCKET_NAME].objects[paperclip_file_path].copy_from(relative/temp/location/file.jpg)

thanks to https://github.com/uberllama

Clinkerbuilt

Thank you for the answer. May I ask you how you proceed to monkey patch this ? Do you make a initializer like paperclip_holder.rb to leave the gem intact ? – Pullet

You can override the method by creating an initializer that uses the class_eval method. I will add an example to my answer. – Clinkerbuilt

@user3181644 - I added a fuller explication to my answer – Clinkerbuilt

That's very helpful, since it's the first time I use this kind of hack. However, after adding the modified storage module inside a decorator, and copying the file in an before_save (because form is on create+update actions) callback, I can observe in my logs that : pushing the form submit button upload the original file (it shouldn't), and then, the background job process the styles (like desired). Paperclip reads the decorator but doesn't seem to block the original style unfortunately – Pullet

@user3181644 - My understanding is that your workflow is something like this: 1)Browser uploads image to AWS S3 2)Paperlip downloads image to /tmp directory of server 3)Paperclip processes image generating styles 4)Paperclip uploads styles to their final location on S3 5) AWS transfers original from temp location to final location. Is this accurate? At what point and in what way is the original being uploaded incorrectly according to your logs? – Clinkerbuilt

You are totally true. I dropped the "and @queued_for_write[:your_processed_style].present?" condition and now the original is blocked. However, that's another problem for me since models (logo, avatars, ...) doesn't have the same :styles and some shouldn't have the :original blocked. I quite confused, wondering if my logic is good or if I missing something regarding paperclip use with bg jobs. I can post my model callbacks if that helps. Thank you for all your time. – Pullet