P8

P8
Installed	June 2016
Operating System	Linux (RHEL 7.2) le
Number of Nodes	2x ( Power8 with 2x NVIDIA K80)
Interconnect	Infiniband ( EDR )
Ram/Node	512 GB
Cores/Node	2 x 8core (16 physical, 128 SMT)
Login/Devel Node	p8t01, p8t02
Vendor Compilers	xlc/xlf

Specifications

The P8 Test System consists of two IBM Power 822LC Servers each with 2x8core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 16 physical cores to support up to 128 threads. Each node has two NVIDIA Tesla K80 GPUs with CUDA Capability 3.7 (Kepler), consisting of 2xGK210 GPUs each with 12 GB of RAM, so each node has effectively 4 GPU's.

Compile/Devel/Test

First login via ssh with your scinet account at login.scinet.utoronto.ca, and from there you can proceed to p8t01 and p8t02.

Software

GNU Compilers

gcc version 4.8.5 is the default with RHEL 7.2. To load the newer advance toolchain version use:

module load gcc/5.3.1

IBM Compilers

IBM compilers xlc/xlc++ 13.1.4 and xlf 15.1.4 are available by default

NVIDIA toolkit

Driver Version

The current NVIDIA driver version for gravity is 352.93

CUDA

The current installed CUDA Tookit is 7.5.

module load cuda/7.5

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

/usr/local/cuda-7.5/

OpenMPI

Currently OpenMPI has been setup on the two nodes connected over FDR Infiniband.

$ module load openmpi/1.10.3-gcc-4.8.5 
$ module load openmpi/1.10.3-gcc-5.3.1
$ module load openmpi/1.10.3-XL-13_15.1.4

PE

IBM's Parallel Environment (PE), is available for use with XL compilers using the following

$ module pe/xl.perf

mpiexec -n 4 ./a.out

documentation is here