MPI Implementation

The Message Passing Interface (MPI) implementation is an important part of an HPC cluster. The vast majority of software that needs to communicate across nodes uses MPI. Although it is not the only means of communication between processes it is the only one that we provide a standard implementation for that can be used both for compiling and running software. Other more specialized communication implementations are typically built with the software that utilizes them.

More information about MPI can be found at the following locations:

There are several implementations of the MPI standard. The one that we use on the campus HPC systems at the University of Iowa is OpenMPI.

There are several implementations of OpenMPI on the HPC systems to account for different compilers, debug, etc. To see what is available

[gpjohnsn@argon-login-2 ~]$ module -r avail ^openmpi

------------------------------------------------------- /opt/modules --------------------------------------------------------
   openmpi/1.10.7_gcc-4.8.5                         openmpi/2.1.2_cuda-8.0.61_gcc-4.8.5
   openmpi/1.10.7_gcc-5.4.0                         openmpi/2.1.2_cuda-9.0.176.1_gcc-4.8.5
   openmpi/1.10.7_parallel_studio-2017.4            openmpi/2.1.2_cuda-9.1.85.2_gcc-4.8.5
   openmpi/1.10.7_pgi-18.4                          openmpi/2.1.2_gcc-4.8.5
   openmpi/2.0.1_cuda-8.0.44_gcc-4.8.5              openmpi/2.1.2_gcc-5.4.0
   openmpi/2.0.1_cuda-8.0.61_gcc-4.8.5              openmpi/2.1.2_memcheck_cuda-8.0.61_gcc-4.8.5
   openmpi/2.0.1_gcc-4.8.5                          openmpi/2.1.2_memcheck_cuda-9.0.176.1_gcc-4.8.5
   openmpi/2.0.1_gcc-5.4.0                          openmpi/2.1.2_memcheck_cuda-9.1.85.2_gcc-4.8.5
   openmpi/2.0.1_memcheck_cuda-8.0.44_gcc-4.8.5     openmpi/2.1.2_memcheck_gcc-4.8.5
   openmpi/2.0.1_memcheck_cuda-8.0.61_gcc-4.8.5     openmpi/2.1.2_memcheck_gcc-5.4.0
   openmpi/2.0.1_memcheck_gcc-4.8.5                 openmpi/2.1.2_memcheck_parallel_studio-2017.1
   openmpi/2.0.1_memcheck_gcc-5.4.0                 openmpi/2.1.2_memcheck_parallel_studio-2017.4
   openmpi/2.0.1_memcheck_parallel_studio-2017.1    openmpi/2.1.2_memcheck_pgi-18.4
   openmpi/2.0.1_memcheck_parallel_studio-2017.4    openmpi/2.1.2_parallel_studio-2017.1
   openmpi/2.0.1_parallel_studio-2017.1             openmpi/2.1.2_parallel_studio-2017.4            (D)
   openmpi/2.0.1_parallel_studio-2017.4             openmpi/2.1.2_pgi-18.4

  Where:
   D:  Default Module

Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Once you have loaded the appropriate environment module then the mpirun command will be available for running your MPI programs. For building MPI programs there are compiler wrappers. This is why there are different instances of openmpi for different compilers. Using the compiler wrappers will make it easier to build your MPI code as it will handle the include and library paths for you. For example, after loading the openmpi module, you can see what the wrapper provides with the -showme option.

[gpjohnsn@argon-login-2 ~]$ module load openmpi/2.1.2_parallel_studio-2017.4
[gpjohnsn@argon-login-2 ~]$ mpifort -showme
ifort -I/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/include -I/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib -Wl,-rpath -Wl,/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib -L/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

That can be broken down into components if desired

[gpjohnsn@argon-login-2 ~]$ mpifort -showme:command
ifort
[gpjohnsn@argon-login-2 ~]$ mpifort -showme:compile
-I/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/include -I/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib
[gpjohnsn@argon-login-2 ~]$ mpifort -showme:link
-I/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib -Wl,-rpath -Wl,/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib -L/opt/apps/openmpi/2.1.2_parallel_studio-2017.4/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

So when an mpi compiler wrapper is used the proper libraries are linked in for you when the wrapper is set as the compiler.

Beginning with openmpi-1.7, the mpif77 and mpif90 compiler wrappers are deprecated for the mpifort compiler wrapper.

Note that there are some applications that provide a version of MPI, either OpenMPI or MPICH, in the project source code tree. Two examples of this are Petsc and OpenFoam. Unless you really know what you are doing you should not build or use the MPI implementations included in-tree of any application. There are several reasons

MPICH does not support OmniPath so building and using that will lead to very poor performance with MPI applications as all of the communication will occur over 1Gbps Ethernet as opposed to 100Gbps OmniPath on Argon. In addition, OmniPath can employ RDMA (Remote Direct Memory Access) which standard Ethernet can not.
The MPI implementation may wind up getting built without support for the SGE (Sun Grid Engine) queue system. If that happens then the MPI implementation will not transfer process control to SGE and will lead to stuck processes if a job fails. This affects everyone who may use those nodes and often leads to mysteriously poor performance until the problem is identified.

To insure the best performance and compatibility use the OpenMPI implementation with the compiler wrappers provided by us by loading the appropriate environment module.