We run one consumer per python app and run multiple processes based on the queue load. As a next step, I would like to auto scale these consumer processes so that we can avoid unnecessary processes when the queue size is low and add additional processes when the pending messages grow.
How do people handle such an auto scaling mechanism for consumers, if anyone does?
Thanks.