Phi
| Intel Xeon Phi / NVIDIA Tesla K20 | |
|---|---|
| Installed | April 2013 |
| Operating System | Linux Centos 6.4 |
| Number of Nodes | 1 |
| Interconnect | DDR Infiniband |
| Ram/Node | 32 GB |
| Cores/Node | 8 with Xeon Phi & K20 |
| Login/Devel Node | gravity01 |
| Vendor Compilers | nvcc,pgcc,icc,gcc |
| Queue Submission | none |
This is a single test node, for investigating new accelerator technologies. It consists of a single x86_64 node with one 8-core Intel Sandybridge Xeon E5-2650 2.0GHz CPU with 32GB of RAM. It has a single NVIDIA Tesla K20 GPU with CUDA Capability 3.0 (Kepler) with 2496 CUDA Cores and 5 GB of RAM as well as a single Intel Xeon Phi 3120A with 57 1.1 GHz cores and 6GB of RAM. The node is interconnected to the rest of the clusters with DDR Infiniband and mounts the regular SciNet GPFS filesystems.
Login
First login via ssh with your scinet account at login.scinet.utoronto.ca, and from there you can proceed to gravity01.
Queue
As this is a single node users are expected to use it in a "friendly" manner as this system is not setup for production usage, and primarily for investigating new technologies run times are limited to under 4 hours. To access the node you need to use the queue, similar to the standard ARC and GPC compute nodes, however with a maximum walltime of 4 hours.
For an interactive job use
qsub -l nodes=1:ppn=8,walltime=1:00:00 -q arcX -I
Software
The same software installed on the GPC is available on arcX using the modules framework. See here for full details.
NVIDIA Tesla K20
See the Gravity wiki page for full details of the available CUDA and OpenCL compilers and modules. To use all the K20 (Kepler) features a minimum of CUDA 5.0 is required.
CUDA
module load gcc/4.8.1 cuda/6.0
Here, gcc is loaded because it is a prerequisite of the cuda module.
You'll have to let the cuda compiler know about the capabilities of the Fermi graphics card by supplying the flag -arch=sm_13 or -arch=sm_20.
Driver Version
The current NVIDIA driver version for the K20 is 340.32
Xeon Phi
Compilers
The Xeon Phi uses the standard intel compilers, however requires at least version 13.1
module load intel/14.0.0
MPI
IntelMPI also has Xeon Phi support
module load intelmpi/4.1.1.036
NOTE: Be sure to use mpiifort for compiling native MIC Fortran code as the mpif77,mpif90 scripts ignore the -mmic flags and will produce host only code.
Tools
The Intel Cluters Tools such as vtune amplifier and inspector are available for the Xeon Phi by loading the following modules.
module load inteltools
OpenCL
OpenCL version 1.2 is available for the Xeon Phi on arcX
/opt/intel/opencl
Direct Access
The Xeon Phi can be accessed directly from the host node by
ssh mic0
The host node arc09 mounts the standard SciNet filesystems, i.e. $HOME and $SCRATCH, however to share files between the host and Xeon Phi use /localscratch/$HOME which shows up as $HOME on "mic0".