Heroku timeout when uploading big file to s3

Question

I got a problem with my application based on RefineryCMS 1.0.8 deployed on heroku.

Heroku have a request timeout of 30 seconds. When uploading a file > 4MO through the RefineryCMS admin, it will sometimes take more than 30 seconds to perform the request (mostly because of the time needed to upload the file to s3).

In this case, the dyno will be "frozen" and unresponsive for around 30 minutes (every request during this time end up by a H12 Error request timeout). This behaviour is exactly what describe christos in the last message in this discussion

From this other discussion and the heroku documentation : "Direct Upload to S3 is the preferred approach if you’re working with file uploads bigger than 4MB. The idea is to skip the hop to your dyno, making a connection from the end user browser directly to S3."

The direct upload to S3 should be the solution but it looks difficult for me to implement it properly with RefineryCMS / Dragonfly / Fog. I'm not sure if it's possible without big modification in theses tools and I'm wondering if there is an alternative.

Thanks for your help

Rotor

Answer 1

Not really. The Heroku timeout is a set in stone thing that you need to work around. Direct upload to S3 is the only option, with some sort of post-upload processing required.

There are solutions out there such as CarrierWaveDirect, but I don't know how easy this would be to use with Refinery.

Aqua

It's not really set in stone. If you use an evented solution (e.g., Eventlet), you can set the timeout higher (I just set mine to 60 seconds and spent 48 seconds successfully uploading a photo to my application), and Heroku won't care, because it's non-blocking. – Ostiole

Answer 2

You could try to add unicorn to your application.

config/unicorn.rb

worker_processes Integer(ENV["WEB_CONCURRENCY"] || 3)

timeout 180 # << Unicorn Timeout will allow a longer upload time.

preload_app true

before_fork do |server, worker|

https://devcenter.heroku.com/articles/rails-unicorn

Shaman

Does this override the heroku timeout? – Baby

When using rails this allowed for a 180 second time out that allowed my s3 upload to finish - I think Heroku defaults to what ever timeout is set in unicorn. – Shaman

Heroku router does not default to whatever is set in the app unicorn config. – Dissimilitude