Scaling in Rabbitmq

Question

While using Clusters in Rabbit MQ, I was planning to use the Competing Subscriber pattern.

Producer : 1
Exchange : 1 direct
Queue : 1
Consumers : n (multiple) listening to the same queue.

Now if I have a cluster containing 3 nodes, there would be but one queue from where the consumer gets the message (irrespective of number of nodes in cluster). If I have to scale up with the above constraints, do you think I should be having multiple clusters.

Please advise.

Second related question, how clustering would help in scalablity?

Verticillaster

Please read this: groups.google.com/forum/#!topic/rabbitmq-users/2gxCqN_NHhs I think could help you. – Legitimate

Answer 1

Clustering is not there to improve Scalability (although it might help in some situations) but to improve High Availability.

For Scalability see the Sharding Plugin which can be used together with clustering https://github.com/rabbitmq/rabbitmq-sharding

Note that you need to find out if said plugin meets your requirements, the README tells of the limitations of the plugin

Environment

Answer 2

Generally speaking, clustering in RabbitMq helps with scalability and high availability. It is important to make a distinction between replication (mirrored queues, high-availability,...) and distribution of different queues between different nodes. There is a great blog post about Clusters & scalability.

To answer the OP question, you can't scale up your configuration as is. Having a single queue usually means that you want messages in it to be processed in order. At the same time, you are mentioning that you have multiple parallel consumers which might suggest (not necessarily) that you want to parallel process your messages. Assuming it is indeed parallel processing you're after, then you should split the queue into multiple queues that will then be distributed (scaled out) in the cluster and increase throughput. You can also use rabbitmq-sharding which does a similar thing under the hood but is not necessarily more convenient (ref).

And finally multiple clusters is a big no for this case (for most cases). It is synonym to deploying two completely independent messaging services.

Merell