Main differences between threading with shared memory and MPI?

Question

although I have been playing with pthreads, OpenMP, intel TBB, and threading in general for a while I still don't understand what is the main difference between a message passing interface implementation like OpenMP and a classic threading library it's still unclear to me.

Assuming that writing all the boilerplate code for a threading pool it's not a problem in my case, and I'm using C++, the difference between this 2 technologies boils down to ... ?

I'm also interested in operating with threads over the network while distributing tasks to all the connected machines.

Right now I'm also not considering the limitations in terms of number of platforms supported by OpenMP/OpenMPI because I would like to understand just how this 2 concepts work.

Argyll

message passing interface implementation like OpenMPI ... – Ria

Answer 1

As a complement to Mike Seymour answer :

The main trade-off depends on what you have to share between your process and threads. With shared memory, you actually share the data between the execution contexts.

With messaging, you need to copy the data to pass it between the execution contexts (threads, processes, processes over several computers).

If your data is small (read: data transmission time is small) compared to the execution time of your context, then MPI should not have a significant overhead compared to shared memory.

At the opposite, if the data to be shared is large (data transmission time) is of the same order of magnitude compared to your execution time, then MPI may not be a good idea.

Last, it you want to cross the boundaries of a single computer, shared memory is out of the game.

Lonely

shared memory across multiple computers is possible with specialized hardware (shared reflective memory) and libraries... Depending on the application's purpose, you can create requirements that certain software requires certain hardware – Comate

so in the MPI I don't pay a fee for a change of context ? Does the MPI provides a protocol to keep the workers in sync and finish the task ? I mean a protocol to handle things like a machine that goes offline or a bad result from a worker and things like that that usually happen in a network-related environment. – Argyll

@g19fanatic: True. If was answering in the context of standard (consumer grade) hardware. – Lonely

@user2485710: I don't know OpenMPI sufficiently to answer this point. My point is about the major distinction between shared memory and messaging architectures. – Lonely

The commercial ScaleMP vSMP provides a virtual Distributed Shared Memory platform that runs on pretty much commodity hardware (as long as one can consider InfiniBand commodity networking). Intel Cluster OpenMP also provided a way to run OpenMP on distributed systems with very minimal code changes. Unfortunately Intel killed the project. – Friedlander

Answer 2

"Classic" threading shares all memory between the threads. This is rather dangerous, since it's very easy to accidentally modify data that another thread might be using, leading to nasty bugs. The onus is on the programmer to carefully protect data against unsafe access. This also (usually) requires all processes to be running on the same machine, with access to the same physical memory.

Using independent processes with a message-passing interface gives you more control over which data is shared and which is private to each process; there is little or no danger of one process unexpectedly modifying another process's state. And as you say, the message passing interface can be generalised to pass messages across a network between processes on separate machines.

Pupa

and what about performance vs portability vs costs ? what's the tradeoff ? – Argyll

@user2485710: It depends heavily on what you're doing and what resources you're using to do it. I couldn't cover the trade-offs here in any meaningful way. – Pupa

but that is exactly the core of what I'm asking, where I can find this informations ? – Argyll