MSMQ messages always arrive delayed by exactly 3 minutes on the same machine

Question

I'm facing an extremely puzzling problem. I have a Windows service that monitors two MSMQ queues for input and sends messages to another MSMQ queue. Although the send operation seems instant from the service's perspective it actually takes the message exactly three (3) minutes to arrive (as shown in the properties window in the MSMQ MMC). I've been testing this problem with nothing else listening on the other side so that I can see the messages piling up. This is how the service sends messages:

var proxyFactory = new ChannelFactory<IOtherServerInterface>(new NetMsmqBinding(NetMsmqSecurityMode.None)
{
    Durable = true,
    TimeToLive = new TimeSpan(1, 0, 0),
    ReceiveTimeout = TimeSpan.MaxValue
});

IOtherServerInterface server = this.proxyFactory.CreateChannel(new EndpointAddress("net.msmq://localhost/private/myqueue"));

var task = new MyTask() { ... };
using (TransactionScope scope = new TransactionScope(TransactionScopeOption.Required))
{
    server.QueueFile(task);
    scope.Complete();
}

The service is running on Windows Server 2008 R2. I also tested it on R1 and noticed the same behavior. Again, everything happens on the same machine. All components are deployed there so I don't think it could be a network issue.

EDIT #1:

I turned on the WCF diagnostics and what I noticed is very strange. The MSMQ datagram does get written normally. However, after the "a message was closed" trace message there is nothing going on. It is as if the service is waiting for something to happen. Exactly 3 minutes later and exactly when the MSMQ message arrives (according to the MSMQ MMC), I see another trace message about a previous activity. I suspect there is some kind of interference.

Let me give you more details about how the services work. There is an IIS app which receives tasks from clients and drops them in an MSMQ queue. From there, the troublesome service (MainService) picks them up and starts processing them. In some cases, another service (AuxService) is required to complete the task so MainService sends a message (that always gets delayed) to AuxService. AuxService has its own inbox queue where it receives MSMQ messages and when it's done, it sends an MSMQ message to MainService. In the meanwhile, the thread that sent the message to AuxService waits until it gets a signal or until it times out. There is a special queue where MainService looks for messages from AuxServices. When a message is received the abovementioned thread is woken up and resumes its activity.

Here's a representation of the whole architecture:

• IIS app -> Q1 -> MainService
• MainService -> Q2 -> AuxService
• AuxService -> Q3 -> MainService

Although all operations are marked with OneWay, I'm wondering whether starting a MSMQ operation from within another MSMQ operation is somehow illegal. It seems to be the case given the empirical evidence. If so, is there away to change this behavior?

EDIT #2:

Alright, after some more digging it seems WCF is the culprit. I switched both the client code in MainService and the server code in AuxService to use MSMQ SDK directly and it works as expected. The 3 minute timeout I was experiencing was actually the time after which MainService gave up and considered that AuxService failed. Therefore, it seems that for some reason WCF refuses to perform the send until the current WCF activity exits.

Is this by design or is it a bug? Can this behavior be controlled?

Rutherford

Do you use authentication? Sometimes these delays are related to networking/infrastructure problems related to authentication. – Pentagram

I don't use any authentication. Moreover, 'Everyone' has full access to the queue. – Rutherford

A send operation to a local queue is instant as no outgoing queue is required. If you used Performance Monitor to track MSMQ, I would expect you to see the message being delivered when it is sent. If not, then it may be a delay waiting for the transaction to commit. Easy7 test would be to send to a non-transactional queue and see if the same delay occurs. – Lehmbruck

@JohnBreakwell: I checked and the behavior is the same. – Rutherford

Can we narrow this down further? Are you saying it takes 3 minutes between a message arriving in Q1 and MainService putting a message in Q2? – Lehmbruck

I did. I feel like I'm slowly zeroing in on the root cause. The 3 minute delay is between MainService finishing sending the message to AuxService and it timing out. Only after this timeout does the message actually appear in the queue. – Rutherford

Answer 1

You have transactions setup on the queue code, do you have the msmq object setup for transactions? 3 minutes sounds like the timeout period for a Distributed Transaction Coordinator enlistment.

Darrin