What is the recommended maximum size for a Sidekiq job payload?

Question

What would be the recommended maximum size for a job payload?

As a specific example, is a HTML document comprised of 500kb to 1Mb of text too large to be passed in to a job payload?

Since Sidekiq is backed by redis I'd say 512Mb, but I wonder if there's a limitation on the Sidekiq side of things.

Friedlander

That would add a bit of serialization and networking overhead. It'll work but I have no idea how much overhead it will entail. You'll need to benchmark it. – Meave

Thanks for your comment Mike. I just wanted to know if there was a maximum value imposed by Sidekiq or if there's a maximum value in which Sidekiq would start misbehaving. Since there isn't one, there's no thing left to do but benchmarking it. – Friedlander

@jdscosta91, would you please share your findings? I'm interested as well. – Infrastructure

Answer 1

When you call perform_async Sidekiq makes a Hash of the arguments, worker class, and other information the job will need. Then it serializes the Hash as JSON (collectively the "payload") and pushes it into Redis with lpush. See A Tour of the Sidekiq API by the author of Sidekiq for more.

The only limits are those of Redis and Ruby's string sizes, and your Redis memory. For a Ruby String that's 2G if you're 32bit and you'll-run-out-of-memory-first for 64 bit.

For Redis that is 512M. Note that's 512M after the content is serialized as JSON. If it's mostly text this will be a small amount. If it's binary data, for example if you compress the text, it could be signficiantly larger.

What would be the recommended maximum size for a job payload?

As small as possible. Large payloads require expensive JSON serialization and de-serialization and consume both Redis and worker memory risking out of memory errors.

Instead of sending the content of a file, store the file somewhere the worker can access. This could be a shared disk, or an S3 Bucket. Send only what is needed for the worker to retrieve the file.

See Best Practices in the Sidekiq wiki.

Crosscut

Answer 2

See this article, you should make your job parameters small and simple. Just store some simple identifiers, and then Look up the objects once you actually need them in your perform method.

And Because it need serialization and deserialization, it will be extra cost to you add the html content to job. So just save the html-content into string or some container and send the string id or container id to redis for efficiency and simplity.

Moreta

Thanks for your answer GuagshengZue. I'm aware of the best practices and of the the cost/benefits of having big/small payloads, I just want to know if there is some imposed limitation or not, or if there's a maximum value in which Sidekiq would start misbehaving. – Friedlander

there is not imposed job size limit of sidekiq. – Moreta