What is the difference and relationship between mpirun, mpiexec and mpiexec.hydra?

Question

I got confused about 3 things: mpirun, mpiexec and mpiexec.hydra

On my cluster, all of them exist, and all of them belong to intel.

What is the difference and relationship between them? Especially, what on earth is mpiexec.hydra? Why there is a dot between mpiexec and hydra, what does it mean?

Synthesize

Possible duplicate of What's the best advance of Hydra MPI – Watchtower

@JohnZwinck Hi, JohnZwinck. I don't think so. I know that post, but after reading it, I still don't understand. So I made this post – Synthesize

Answer 1

mpirun and mpiexec are basically the same - the name of the process launcher in many MPI implementations. The MPI standard says nothing about how the ranks should be started and controlled, but it recommends (though does not demand) that, if there is a launcher of any kind, it should be named mpiexec. Some MPI implementations started with mpirun, then adopted mpiexec for compatibility. Other implementations did the reverse. In the end, most implementations provide their launcher under both names. In practice, there should be no difference in what mpirun and mpiexec do.

Different MPI implementations have different means of launching and controlling the processes. MPICH started with an infrastructure called MPD (Multi-Purpose Daemon or something). Then it switched to the newer Hydra process manager. Since Hydra does things differently than MPD, the Hydra-based mpiexec takes different command-line arguments than the MPD-based one and to make it possible for users to explicitly select the Hydra-based one, it is made available as mpiexec.hydra. The old one is called mpiexec.mpd. It is possible to have an MPICH-based MPI library that only provides the Hydra launcher and then mpiexec and mpiexec.hydra will be the same executable. Intel MPI is based on MPICH and its newer versions use the Hydra process manager.

Open MPI is built on top of Open Run-Time Environment (ORTE) and its own process launcher is called orterun. For compatibility, orterun is also symlinked as mpirun and mpiexec.

To summarise:

• mpiexec.something is a specific version of the MPI process launcher for a given implementation
• mpiexec and mpirun are the generic names, usually copies of or symbolic links to the actual launcher
• both mpiexec and mpirun should do the same
• some implementations name their launcher mpiexec, some name it mpirun, some name it both, and that is often the source of confusion when more than one MPI implementations are simultaneously available in the system paths (e.g. when installed from distro packages)

Yawning

Thank you so much, Hristo Iliev. Your answer makes things much clearer. And it seems that mpirun must specify -n options, while mpiexec and mpiexec.hydra doesn't have to be – Synthesize

On my Knights Landing system with the Intel compiler mpiexec is a symbolic link to mpiexec.hydra (i.e. they are the same). However, mpirun is a shell script which calls mpiexec.hydra. – Foretopsail

What does your sentence "both mpiexec and mpirun should do the same" mean? You mean they should do the same thing? – Foretopsail

It means both are launching an MPI job in one way or another. – Yawning