Is message priority inherently unimportant in message queue systems?

Question

It seems like most of the messaging systems I've looked at have basic, if any, support for priority message queues. For example, the AMQP only specifies a minimum of 2 priorities. RabbitMQ, an AMQP implementation, doesn't support any priorities. ActiveMQ will be getting support for 10 message priorities in version 5.4 in a couple days. 10 priority levels is the specified by the JMS spec.

A priority queue in the non-messaging sense of the word orders its contents based on an arbitrary field with an unconstrained range of priorities. Why does an implementation like this not exist as part of a messaging system? As I asked in the title, is priority an inherently non-messaging concept?

I realize that one answer might be that the concept of priority introduces the possibility of messages infinitely languishing in the queue while higher priority messages are processed. Are there other reasons?

Alcheringa

A couple of comments: This might be better suited to CSTheory.SE, I think that you nailed the priority problem in your post. You have to have a way to prevent deadlocking or languishing... I tend to think of a message queue as having the purpose of forwarding messages, not of being responsible for any primary business logic. – Arty

Answer 1

In general, message queue systems are used to ensure delivery of messages between disparate systems.

Usually, there is some sort of once-and-only-once guarantee, and often a further promise that the messages will come in order.

By and large, that then informs the design of the system(s) that you are building and hooking together.

The concepts of priority between decoupled systems often don't make that much sense.

That said, one common workaround is to have two queues, one high priority and one background priority. The inherent problem is then made clear however, because of course the receiving system probably cannot halt processing the low-level request when a higher priority request comes in, so they are generally done sequentially at that level of granularity.

Tanta

Thanks for the answer. This is how I feel about it too. My initial feeling of discontent stems from the fact that I was trying to implement a job queueing system with a message queue, which I realize now was incorrect. – Alcheringa

Answer 2

BTW ActiveMQ now supports priority messaging in 5.4.x via JMSPriority headers.

Rather than getting the message broker to re-order messages within some buffer as they arrive, there are often better techniques to implement priority consumption, such as having a dedicated consumer pool for high priority messages. Then irrespective of how much noise there is from low priority messages, high priority messages will always get though.

Given the asynchronous nature of messaging its easy to fill up buffers, network pipes and prefetch queues with low priority messages if using things like JMSPriority headers etc.

Vennieveno

Answer 3

It seems to me that the idea is probably more akin to "process priority" than the priority values in a priority queue. Certainly that is consistent with the sentence-or-two about it in the JMS spec, and evidently with the AMQP spec as well.

Hoodwink

Answer 4

One has to be careful in that too many priorities aren't used to the point where using the program becomes more burdensome than going through each message.

Gorges

Answer 5

Messaging systems are designed and optimized for chronological ordering. File systems are optimized for appending files and not inserting data at the beginning or in the middle. Queue-like data structures are usually optimized for append at the end and removal from the head. For file systems, this means appending to file (adding) and appending to a transaction log (removal), and deleting message files once they are consumed (removal).

Introducing priorities to a processing queue effectively turns the queue into a data structure that has both chronological and priority sorting. Basically, when it comes to working with file storage, it's quite sub-optimal as you have to create some sort of indexing strategy.

Tilden