Azure Queue body is too large and exceeds the maximum permissible limit

Question

I am getting the following error with my storage queue (NOT BLOB - I know others have seen this with the blob) when pushing the message from my c++ app to Azure Storage:

The request body is too large and exceeds the maximum permissible limit.

I'm aware that I probably need to cut the json down, but are there any other suggestions? (as in increase message size somewhere?)

Mealy

Answer 1

As others have stated, the Azure Storage Queue message size limit (64K) is a hard limit.

Aside from encoding, compression (minification), etc: The most common pattern to work around this limit is to not store your payload in the queue message; rather, store it in something like Blob storage, and only store the message type & metadata (if needed), along with a URI pointing to the blob containing your payload to process.

By following this pattern, and using blob storage for your payload, you effectively have potential payload size of 4+ TB. And you also have the ability to persist your payload if needed (whereas queue messages are deleted after processing).

Trouper

I found an example with storage queue and blob Dealing with large Service Bus messages using claim check pattern – Malnutrition

Answer 2

The maximum size for a message in the Azure storage queue is 64KB (48 KB when using Base64 encoding) based on the latest Azure Storage Service Limits documentation as below.

It is non-configurable and at the moment Azure support also will not increase the size upon request.

https://learn.microsoft.com/en-us/azure/azure-subscription-service-limits#storage-limits

I will suggest you to reduce the size of your JSON message e.g. JSON minify

Catboat

Adding one more thing: Since Azure uses UTF-16 encoding to store the data (i.e. use 2 bytes to store each character so to speak), message size is essentially limited to 32KB. – Trapezohedron

Hi @GauravMantri, is there any documentation which mention Azure storage queue message is using UTF-16 encoding? Thanks – Catboat

I remember reading it somewhere but I am not able to find that. However I have tried it myself and for any string/byte array greater than 32KB in size I got this error. – Trapezohedron

Is this the reason when I use the Azure Portal to put a decently large JSON message in a queue, the checkbox “Encode the message body in Base64” gets grayed out? I am not sure how much 32KB is in number of characters including spaces. – Adamsen

WAIT WHAT?! Max size is 48KB when using Base64 encoding? Why have I never seen that documented anywhere? Wow - awesome info dude. Solved my problem! – Lw