GPC MPI Versions

You can only use ONE mpi version at a time as most versions use the same names for the mpirun and compiler wrappers, so be careful which modules you have loaded in your ~/.bashrc.

OpenMPI

To use OpenMPI compiled with the intel compilers load the modules

module load intel openmpi

or for the gcc version use

module load gcc openmpi/1.4.2-gcc-v4.4.0-ofed 

The MPI library wrappers for compiling are mpicc/mpicxx/mpif90/mpif77.

OpenMPI has been built to support various communication methods and automatically uses the best method depending on how and where it is run. To explicitly specify the method you can use the following --mca flags on ethernet

mpirun --mca btl self,sm,tcp -np 16 -hostfile $PBS_NODEFILE ./a.out

and the following for infiniband

mpirun --mca btl self,sm,openib -np 16 -hostfile $PBS_NODEFILE ./a.out

and the following for IPoIB

mpirun --mca btl self,sm,tcp --mca btl_tcp_if_exclude lo,eth0 -np 16 -hostfile $PBS_NODEFILE ./a.out

For mixed openMP/MPI applications, set OMP_NUM_THREADS to the number of threads per process and add '--bynode' to the mpirun command, e.g.,

export OMP_NUM_THREADS=4
mpirun -np 6 --bynode -hostfile $PBS_NODEFILE ./a.out

would start 6 MPI processes on different nodes, each with 4 openMP threads. If your script requests 3 nodes, each node gets 2 MPI processes.

For more information on available flags see the OpenMPI FAQ

IntelMPI

IntelMPI is also a MPICH2 derivative customized by Intel. To use IntelMPI (4.x) compiled with the intel compilers load the modules

module load intel intelmpi

The MPI library wrappers for compiling are mpicc/mpicxx/mpif90/mpif77.

IntelMPI requires a .mpd.conf file containing the variable "MPD_SECRETWORD=..." in your $HOME directory. To create this file use

echo "MPD_SECRETWORD=ABC123" > ~/.mpd.conf
chmod 600  ~/.mpd.conf

IntelMPI, like OpenMPI, has been built to support various communication methods and automatically uses the best method depending on how and where it is run.

mpirun -np 16 ./a.out

In contrast to OpenMPI, the above line will only work for intelmpi on the infiniband nodes For the ethernet nodes, you will have to explicit specify the ethernet flags to mpirun.'

To explicitly specify the method you can use the following flags to use ethernet (tcp) and shared memory

mpirun -np 2 -env I_MPI_FABRICS shm:tcp ./a.out

or the following flags for infiniband (dapl) and shared memory (best performance on IB)

mpirun -np 2 -env I_MPI_FABRICS shm:dapl ./a.out

or the following for IPoIB

mpirun -np 2  -env I_MPI_TCP_NETMASK=ib -env I_MPI_FABRICS shm:tcp ./a.out

For hybrid openMP/MPI runs, set OMP_NUM_THREADS to the desired number of OpenMP threads per MPI process and specify the number of MPI processes per node on the mpirun command line with -ppn <num>. E.g.

export OMP_NUM_THREADS=4
mpirun -ppn 2 -np 6 -env I_MPI_FABRICS shm:tcp ./a.out

would start a total of 6 MPI processes each with 4 threads, with each node running 2 MPI processes. Your script should request 3 nodes in this case. Note: to compile for hybrid openMP/MPI with IntelMPI, you need to add the flag -mt_mpi to your compilation command (i.e. mpicc/mpif90/mpicxx).

For large jobs (500+ cores) it is recommended that you use the hydra initialization procedure as apposed to the default mpd boot process that used by default in the "mpirun" statement given in the previous examples.

mpiexec.hydra -hostfile $PBS_NODEFILE -np 512 ./a.out

For more information on these an other flags see Intel's Documentation page especially the "Getting Started (Linux)" Guide.