Why would two processes have an advantage over 2 threads?

Question

I have an MSMQ based location application, where I receive position updates from units in the field and they are processed and put in a database.

The update process does not have dependencies outside the DB, so I my app can be configured with a variable number of threads. As I want the process to be robust under failure, I want to process as much messages as I can, but not more (so if the system fails, I can pick up where I left).

I have the app working correctly, but I've seen that if I raise the number of threads I use to process messages, my avg number of messages is at one level (I use performance counters to measure this), and I get the system to utilize, say, 50% of the CPU time available (I have an Core i7 820QM with 4 physical cores and 8 logical cores), but if I instead of raising threads, launch the same number of processes, I do get to use 100% of the CPU time, and get a much higher number of average events processed.

Can it be a lock contention problem? Something to do with the way Windows 7 treats hyper-threaded processors? I wish to understand the nature of the problem, and any pointers would be really appreciated.

Note: I'm using MSMQ, Rx and Entity Framework in this project.

Stephanotis

how are you raising the number of threads? – Columba

The problem is almost certainly with your usage of Rx. I think Rx coordinates (read takes some locks) your threads in a way that not all cores can be utilized. To get an idea break several times into your app and check in which methods the call stacks are most of the time. That should give you an idea where the locks live. – Chandler

@Wegged: new Thread(() => { ... } { IsBackground = true }.Start(); – Stephanotis

I'm seeing now that I have been using more threads than I thought. Each message that's received from the MessageQueue by the BeginReceive/EndReceive pair is using a thread for it's reception, and I'm processing my message in that thread (not what I thought and wanted). Still the fact that doubling the threads (irrespective of that threads creating yet more threads) is slower than doubling the processes. Contention it is I guess, but I would rather know where the problem lies. My profiler runs, put the time in my update methods, not in the frameworks. – Stephanotis

Answer 1

Without seeing your code its hard to give an exact answer but I'll add some possibilities anyway

Firstly contention can have a huge impact, what locking types are you using ?

Secondly .Net framework can have an impact, for .Net 2.0 & earlier kernel objects were used as the locking mechanism. These required kernel transitions to change state which impacted performance.

Ipoh

I'm using .Net4. My locking strategy is basically, no locks managed by me. I receive messages from MSMQ in an Rx observable using FromAsyncPattern and the BeginReceive, EndReceive methods. Each thread has it's own copy of this observable wich refers to it's own MessageQueue object. Basically, I share nothing, so I take no locks, but still can't ramp up my utilization by using more threads, but I can by using Processes. There probably are things that are shared and have locks in them that I can't see (EntityFramework or SqlConnection perhaps? I'll try disabling connection pooling!). – Stephanotis

After I've seen that my processing was not being done in the correct threads, I've changed some things and ensured that was the case. As I ramped up my threads, the number of contentions reported was getting higher and higher, making it difficult for those threads to progress. As the source of the contention is inside library code (and out of my reach), I'll just switch to using processes instead of threads for this scenario. Thank you! – Stephanotis