Reading from MSMQ slows down when there is a lot of messages queued

Question

Short introduction

I have a SEDA based system, and used MSMQ for communication (event triggering) between the different applications/services.

One of these services gets messages by file, so I have a file listener that reads the file content and inserts this into a queue (or actually 4 different queues, but that's not very important for the first question).

Server is Windows Server 2008

First question - read slows down

My application that reads these messages at the other side normally reads about 20 messages from the queue per second, but when the service that posts messages start queuing some thousand messages, the read goes down, and the read application only reads 2-4 messages per second. When there is no posting to the queue, the read application can again read up to 20 messages per second.

The code in the reading application is pretty simple, developed in C#, I use the Read(TimeSpan timeout) function in System.Messaging.

Q: Why does the read slows down when there is a lot of messages posted to the queue?

Second question - limitations of TPS

An additional question is about the read itself. It seems there is no difference in how many messages I can read per second if I use 1 or 5 threads to read from the queue. I've also tried implementing a "round robin solution" where the post service are posting to a random set of 4 queues, and the read application had one thread listening to each of these queues, but there is still only 20 TPS even if I read from 1 queue with 1 thread, 1 queue with 4 threads or 4 queues (with one thread per queue).

I know the processing in the thread takes about 50 ms, so 20 TPS is quite correct if there is only one message processed at the time, but the clue with multi threading should be that messages are handled in parallel and not sequential.

There is about 110 different queues on the server.

Q: Why can't I get more than 20 messages out of my queue at the time even with multi threading and the use of several queues?

This is the code running today:

// There are 4 BackgroundWorkers running this function
void bw_DoWork(object sender, DoWorkEventArgs e) 
{
    using(var mq = new MessageQueue(".\\content"))
    {
        mq.Formatter = new BinaryMessageFormatter();

        // ShouldIRun is a bool set to false by OnStop()
        while(ShouldIRun)
        {
            try
            {
                using(var msg = mq.Receive(new TimeSpan(0,0,2))
                {
                    ProcessMessageBody(msg.Body); // This takes 50 ms to complete
                }
            }
            catch(MessageQueueException mqe)
            {
               // This occurs every time TimeSpan in Receive() is reached
               if(mqe.MessageQueueErrorCode == MessageQueueErrorCode.IOTimeout) 
                   continue;
            }
        }
    }

But even if there are 4 threads, it seems all waits for the function to enter the "Receive" point again. I've also tried using 4 different queues (content1, content2, content3 and content4), but still i get 1 message processed every 50 ms.

Does this have anything to do with the TimeSpan in Receive(), and/or is it possible to omit this?

Another question is if the use of private queues, instad of public will solve anything?

Burner

Answer 1

Performance issues.
You don't mention if all the code is running on the server or if you have clients remotely accessing the queues on the server. From the speed, I'll assume the latter.
Also, are the queues transactional?
How large are the messages?

If you want to read a message from a queue, your application does not connect to the queue itself. Everything goes between the local queue manager and the remote queue manager. The queue manager is the only process that writes to, and reads from queues. Therefore having multiple queues or a single queue won't necessarily perform any differently.

The MSMQ queue manager is therefore going to be a bottleneck at some point as there is only so much work it can do at the same time. Your first question shows this - when you put a high load on the queue manager putting messages IN, your ability to take messages OUT slows down. I'd recommend looking at performance monitor to see if MQSVC.EXE is maxed out, for example.

Cene

All code is running on the same server. It's only used to send messages between individual services. It's public queues, and non-transactional. The messages are pretty small. – Burner

Do you need Public queues? Private queues are normally adequate unless you require remote administration. Are you using DIRECT FormatNames to address the queues or pathnames? – Cene

Actually I don't need public queues, but when the system was made I had to because of remote access, but the queues are now just used by local applications. I'm using pathnames to access the queues, but the initialisation of the queue is only done once, so that shouldn't mean too much? – Burner

An easy test would be to sniff the network out from the server to see if there is any extra traffic for queries to Active Directory. – Cene

By the way, 20 messages per second is VERY slow indeed. My desktop machine will receive 10,000 1kb recoverable messages in 32 seconds. That's without any processing of the message - just receiving them from a local queue. – Cene

There shouldn't be any network traffic, as this server also is the AD server. That's exactly what I meant as well, that 20 TPS are extremely little, and I should be able to read much more than that. I wonder if it can have something to do with me using MessageQueue.Receive(TimeSpam timeout), and not only Receive()? – Burner

I'm not a coder so can't comment; your other answerer seems to indicate that there should be a difference. – Cene

Wow, I've been using MSMQ for more than 10 years and I don't remember ever seeing it even break a sweat. This must be a REALLY high-volume scenario. – Wheelwright

Answer 2

Why are you using timespan? - that is a bad thing and here is why.

When developing services and queue you need to program in a theadsafe manner. Each item in the queue will spawn a new thread. Using timespan is forcing each of the threads to use a single timer event thread. These events are having to wait for their turn at the event thread.

The norm is 1 thread per queue events - This is generally your System.Messaging.ReceiveCompletedEventArgs event. Another thread is your onStart event...

20 threads or 20 reads per second is probably correct. Generally when thread pooling you can only spawn 36 threads at a time in .net.

My advice is drop the timer event an make your queue simply process the data.

do something more like this;

namespace MessageService 
{ 

public partial class MessageService : ServiceBase 

{ 

    public MessageService() 

    { 

        InitializeComponent(); 

    } 



    private string MessageDirectory = ConfigurationManager.AppSettings["MessageDirectory"]; 

    private string MessageQueue = ConfigurationManager.AppSettings["MessageQueue"]; 



    private System.Messaging.MessageQueue messageQueue = null; 



    private ManualResetEvent manualResetEvent = new ManualResetEvent(true); 





    protected override void OnStart(string[] args) 

    { 

        // Create directories if needed 

        if (!System.IO.Directory.Exists(MessageDirectory)) 

            System.IO.Directory.CreateDirectory(MessageDirectory); 



        // Create new message queue instance 

        messageQueue = new System.Messaging.MessageQueue(MessageQueue); 



        try 

        {    

            // Set formatter to allow ASCII text 

            messageQueue.Formatter = new System.Messaging.ActiveXMessageFormatter(); 

            // Assign event handler when message is received 

            messageQueue.ReceiveCompleted += 

                new System.Messaging.ReceiveCompletedEventHandler(messageQueue_ReceiveCompleted); 

            // Start listening 



            messageQueue.BeginReceive(); 

        } 

        catch (Exception e) 

        { 



        } 

    } 



    protected override void OnStop() 

    { 

        //Make process synchronous before closing the queue 

        manualResetEvent.WaitOne(); 





        // Clean up 

        if (this.messageQueue != null) 

        { 

            this.messageQueue.Close(); 

            this.messageQueue = null; 

        } 

    } 



    public void messageQueue_ReceiveCompleted(object sender, System.Messaging.ReceiveCompletedEventArgs e) 

    { 

        manualResetEvent.Reset(); 

        System.Messaging.Message completeMessage = null; 

        System.IO.FileStream fileStream = null; 

        System.IO.StreamWriter streamWriter = null; 

        string fileName = null; 

        byte[] bytes = new byte[2500000]; 

        string xmlstr = string.Empty;                

            try 

            { 

                // Receive the message 

                completeMessage = this.messageQueue.EndReceive(e.AsyncResult);                    

                completeMessage.BodyStream.Read(bytes, 0, bytes.Length); 



                System.Text.ASCIIEncoding ascii = new System.Text.ASCIIEncoding(); 



                long len = completeMessage.BodyStream.Length; 

                int intlen = Convert.ToInt32(len);                   

                xmlstr = ascii.GetString(bytes, 0, intlen);                   

            } 

            catch (Exception ex0) 

            { 

                //Error converting message to string                    

            } 

        }

Aubigny

Are you able to reference some documentation for the statement - "Why are you using timespan? - that is a bad thing and here is why." Had a quick look for some doco, but could not find anything. – Bridal