Rails: sporadic Carrierwave/Excon errors

Question

Using carrierwave for our uploaders, we get a couple of Excon errors each week from our production app. For example:

Excon::Errors::BadRequest: Expected(200) <=> Actual(400 Bad Request) excon.error.response :body => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>IncompleteBody</Code><Message>The request body terminated unexpectedly</Message>

We've started wrapping the uploading process in a retry block and it always seems to work fine after another try, but I'm wondering if there is a better solution, as this becomes unwieldy after a while. It seems to me like these errors ought to be handled at a lower level. Is there a better way to handle these issues?

Here's our production configuration:

config.storage = :fog
config.root = Dir.tmpdir
config.cache_dir = 'carrierwave'
config.fog_credentials = {
  provider: 'AWS',
  aws_access_key_id: ENV['AWS_ACCESS_KEY_ID'],
  aws_secret_access_key: ENV['AWS_ACCESS_KEY'],
}

config.fog_directory = ENV['AWS_S3_BUCKET']
config.fog_public = false
config.fog_authenticated_url_expiration = 7.days.to_i

config.enable_processing = true

And we're using gem versions:

fog (1.27.0)
carrierwave (0.10.0)
excon (0.43.0)

Supererogate

Have you managed to fix this? – Sharpshooter

After switching to carrierwave-aws we haven't seen this issue anymore. – Supererogate

Answer 1

It started working again after I wrote my initializer like this, after overcoming a couple of problems, I think AWS endpoints have changed:

CarrierWave.configure do |config|
  config.fog_credentials = {
      :provider               => 'AWS',
      :aws_access_key_id      => ENV['S3_KEY'],
      :aws_secret_access_key  => ENV['S3_SECRET'],
      :endpoint               => "https://s3.amazonaws.com",
      :region                 => ENV['S3_REGION'] 
  }
  config.fog_directory  = ENV['S3_BUCKET']
end

Also, I thought my region was "us-west-2", looking at my AWS administration console, but it only started working when I changed to "eu-west-1".

Later I realized it is not a good idea to specify that endpoint, in fact it is better to leave it in this situation. Anyways, changing to carrierwave-aws as pointed by lobati solved the problem.

Sharpshooter

Were you seeing consistent errors, or were they intermittent? For the most part ours worked, we would just get an error once every day or two. We ended up switching to the carrierwave-aws gem, and haven't seen any errors so far. – Supererogate

Thank you for refereing carrierwave-aws I'm gonna check it. I was getting consistent errors on some images, while others are working fine. I don't understand what is happening because I didn't change anything and it stopped working... I have two projects with the same set up, one is working great, the other one gives me errors like this, always.. – Sharpshooter

Please ignore the previous one. Thank you for referring carrierwave-aws I'm gonna check it. I was getting consistent errors on some images, while others are working fine. I don't understand what is happening because I didn't change anything and it stopped working... I have two projects with the same set up, one is working great, the other one gives me errors like this, always... – Sharpshooter

Also I'm getting some conflicts with the urls. On the project that isn't working properly when I upload something to S3 the url saved on my database is something like "s3.amazonaws.com/<bucket-name>/<path-to-image>", this one gives me a 301 when I try it. Before all this it was saving urls like "s3-eu-west-1.amazonaws.com/<bucket-name>/…>" and they worked. But on the project that I have right now working it is saving urls like this "https://<bucket-name>.s3.amazonaws.com/<path-to-image>" and they also work well. I'll try to figure out what is going on here... – Sharpshooter

Yes, specifying the endpoint was messing with the urls, it is not a good idea. – Sharpshooter

I think fog changed the way that they handle the endpoint recently, so that might have been causing an issue for you. Hopefully carrierwave-aws fixes everything for you. – Supererogate