I need to consume SQS events with my rails application. I've written a Sidekiq job which does a long polling like this:
class SqsConsumerWorker
include Sidekiq::Worker
def perform
...
poller = Aws::SQS::QueuePoller.new(<queue_url>, client: <sqs_instance>)
poller.poll(wait_time_seconds: 20, max_number_of_messages: 10, visibility_timeout: 180) do |messages|
messages.each do |message|
puts message.inspect
end
end
end
end
First problem was when to initiate this job. Currently I've moved the invocation to rails initializer where I've overriden the Sidekiq config.on(:startup) block to call this job. This will help me to start the job on every deployment. (I have also written some logic in this initializer to check the number of workers are not above some limit etc.)
I wanted to understand is there a better way to solve this problem? I've seen the gem shoryuken which abstracts out these things. But I need more control over the consumer and thought of having my own implementation. I also need to understand how to scale up and scale down the number of consumers with this approach.