GPC Quickstart

General Purpose Cluster (GPC)
Installed	June 2009
Operating System	Linux
Number of Nodes	3864 (30,912 cores)
Interconnect	840 nodes 1:1 DDR, 3024 nodes 5:1 QDR
Ram/Node	16 Gb
Cores/Node	8 (16 threads)
Login/Devel Node	gpc01..gpc04 (from login.scinet)
Vendor Compilers	icc (C) ifort (fortran) icpc (C++)
Queue Submission	Moab/Torque

Specifications

The General Purpose Cluster is an extremely large cluster (ranked 16th in the world at its inception, and fastest in Canada) and is where most computations are to be done at SciNet. It is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the first in the world to make use of the new chips). The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,912 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking DDR InfiniBand while the rest of the nodes are connected with 5:1 blocked QDR InfiniBand. The compute nodes are accessed through a queuing system that allows jobs with a minimum of 15minutes and maximum wall time of 48 hours .

Login

First login via ssh with your SciNet account at login.scinet.utoronto.ca, and from there you can proceed to the Development nodes (gpc01, gpc02, gpc03, or gpc04) to compile/test your code.

Compile/Devel Nodes

From a scinet login node you can ssh to gpc01..gpc04. You may also just use /scinet/gpc/bin/gpcdev to take directly to the dev node with lowest cpu load (reassessed every 5 minutes).

% /scinet/gpc/bin/gpcdev -h
  Usage: gpcdev [usual ssh options]
Example: gpcdev -X
redirected to dev node with lowest cpu load

These nodes have the same hardware configuration as most of the compute nodes except with more memory -- 8 Nehalem processing cores with 36GB RAM and QDR Infiniband. You can compile and test your codes on these nodes. To interactively test on more than 8 processors, you can submit an interactive job request.

Your home directory is in $HOME (currently /home/g/group/USER but this can change, so it's best to use the environment variable $HOME or ~ in scripts). You have 10GB there that is backed up. Your home directory cannot be written to by the compute nodes! Thus, to run jobs, you'll use the $SCRATCH directory (currently /scratch/g/group/USER but again, use the environment variable). Here, there is a large amount of disk space, but it is not backed up. Thus it makes sense to keep your codes in /home, compile there, and then run them in the /scratch directory.

Modules and Environment Variables

To use most packages on the SciNet machines - including any of the compilers - , you will have to use the `modules' command. The command module load some-package will set your environment variables (PATH, LD_LIBRARY_PATH, etc) to include the default version of that package. module load some-package/specific-version will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

Note that to use even the gcc compilers you will have to do

$ module load gcc

but in fact you probably should use the intel compilers installed on this system as they often produce faster executables (and occasionally, much faster.)

A list of the installed software is available in Software & Libraries and can be seen on the system by typing

$ module avail

To load a module (for example, the default version of the intel compilers)

$ module load intel

To unload a module

$ module unload intel

To unload all modules

$ module purge

These commands should got in your submission scripts to make sure you are using the correct packages. It is possible to load them in your .bashrc files as well, but this is generally not recommended (see Important .bashrc guidelines), especially if you routinely have to flip back and forth between modules.

Note that a module load command only sets the environment variables in your current shell (and any subprocesses that the shell launches). It does not affect other shell environments; in particular, a queued job that is running is unaffected by you interactively loading a module, and conversely you loading a module at the prompt and then submitting a job does not ensure that the module is loaded when the job runs. To ensure that a module is loaded when a job runs, be sure to put your module load command in your job submission script.

Compilers

The intel compilers are icc/icpc/ifort for C/C++/Fortran, and are available with the default module "intel". The intel compilers are recommended over the GNU compilers. Documentation about icpc is available at http://software.intel.com/en-us/articles/intel-software-technical-documentation/. The Intel compilers accept many of the options that the GNU compilers accept, but tend to produce faster programs on our system. If, for some reason, you really need the GNU compilers, the latest version of the GNU compiler collection (currently 4.4.0) is available by loading the "gcc" module, with gcc/g++/gfortran for C/C++/Fortran. Note that f77/g77 is not supported.

To ensure that the intel compilers are in your PATH and their libraries are in your LD_LIBRARY_PATH, use the command

$ module load intel

Optimize your code for the GPC machine using of at least the following compiler flags:

   -O3 -xHost

(or -O3 -march=native for the GNU compilers).

• If your program uses openmp, add -openmp (-fopenmp for GNU compilers).
• If you get the warning feupdatreenv is not implemented, add -limf to the link line.
• If you need to link in the MKL libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code. Note that this give the link line for the command prompt. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.
• More questions about compiling? See the FAQ.

Debuggers

• ddt - Allinea's graphical parallel debugger, in the ddt module. Highly recommended!
• gdb - The GNU Debugger, available in the gdb module.
• idbc/idb - The intel debuggers, part of the intel module(s).
• ddd - A graphical debuggerThe GNU Debugger, available in the ddd module.

Note that to debug code, you have to give the

-g

flags to the compiler. The intel compiler needs the additional option -debug parallel to debug threaded/OpenMP code.

MPI

SciNet currently provides multiple MPI libraries for the GPC; OpenMPI, and IntelMPI. We currently recommend OpenMPI as the default, as it quite reliably demonstrates good performance on the infiniband network (and did so too on the ethernet network). For full details and options see the complete MPI section.

The MPI libraries are compiled with both the gnu compiler suite and the intel compiler suite. To use (for instance) the intel-compiled OpenMPI libraries, which we recommend as the default (and use for most of our examples here), use

$ module load intel openmpi

in your job submission scripts and on the command-line before compiling. Putting these in your .bashrc is no longer recommended (since Oct 10, 2013).

Other combinations behave similarly.

The MPI libraries define the wrappers mpicc/mpicxx/mpif90/mpif77 as wrappers around the appropriate compilers, which ensure the appropriate include and library directories are used in the compilation and linking steps.

We currently recommend the Intel + OpenMPI combination. However, if you require the GNU compilers as well as MPI, you will want to find the most recent openmpi module available with `gcc' in the version name. This will enable development and runtime with gcc/g++/gfortran and OpenMPI.

For mixed OpenMP/MPI code using Intel MPI, add the compilation flag -mt_mpi for full thread-safety (no such flag is necessary for OpenMPI).

Submitting A Batch Job

The SciNet machines are shared systems, and jobs that are to run on them are submitted to a queue; the scheduler then orders the jobs in order to make the best use of the machine, and has them launched when resources become availble. The intervention of the scheduler can mean that the jobs aren't quite run in a first-in first-out order.

The scheduler will have the job run on one or more of the compute nodes of the GPC. It is important to realize that on compute nodes, your home directory is read-only. You have to run your jobs from the $SCRATCH directory instead. See Data Management for more details on the file systems at SciNet.

The maximum wallclock time for a job in the queue is 48 hours; computations that will take longer than this must be broken into 48-hour chunks and run as several jobs. Also a minimum job length of 15 minutes is enforced and shorter jobs should be batched together. The usual way to do this is with checkpoints, writing out the complete state of the computation every so often in such a way that a job can be restarted from this state information and continue on from where it left off. Generating checkpoints is a good idea anyway, as in the unlikely event of a hardware failure during your run, it allows you to restart without having lost much work.

There are limits to how many jobs you can submit. If your group has a default account, up to 32 nodes at a time for 48 hours per job on the GPC cluster are allowed to be queued. This is a total limit, e.g., you could request 64 nodes for 24 hours. Jobs of users with an LRAC or NRAC allocation will run at a higher priority than others while their resources last. Because of the group-based allocation, it is conceivable that your jobs won't run if your colleagues have already exhausted your group's limits.

Note that scheduling big jobs greatly affects the queuer and other users, so you have to talk to us first to run massively parallel jobs (> 2048 cores). We will help make sure that your jobs start and run efficiently.

If your job should run in fewer than 48 hours, specify that in your script -- your job will start sooner. (It's easier for the scheduler to fit in a short job than a long job). On the downside, the job will be killed automatically by the queue manager software at the end of the specified wallclock time, so if you guess wrong you might lose some work. So the standard procedure is to estimate how long your job will take and add 10% or so.

You interact with the queuing system through the queue/resource manager, Moab and Torque. To see all the jobs in the queue use

$ showq

To submit your own job, you must write a script which describes the job and how it is to be run (a sample script follows) and submit it to the queue, using the command

$ qsub [SCRIPT-FILE-NAME]

where you will replace [SCRIPT-FILE-NAME] with the file containing the submission script. This will return a job ID, for example 31415, which is used to identify the jobs. Information about a queued job can be found using

$ checkjob [JOB-ID]

and jobs can be canceled with the command

$ canceljob [JOB-ID]

Again, these commands have many options, which can be read about on their man pages.

Much more information on the queueing system is available on our queue page.

Batch Submission Script: MPI

A sample submission script is shown below for an mpi job with the #PBS directives at the top and the rest being what will be executed on the compute node.

<source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC
4. PBS -l nodes=2:ppn=8,walltime=1:00:00
5. PBS -N test

1. load modules (must match modules used for compilation)

module load intel openmpi

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. EXECUTION COMMAND; -np = nodes*ppn

mpirun -np 16 ./a.out </source>

The lines that begin #PBS are commands that are parsed and interpreted by qsub at submission time, and control administrative things about your job. In this example, the script above requests two nodes, using 8 processors per node, for a wallclock time of one hour. (The resources required by the job are listed on the #PBS -l line.) Other options can be given in other #PBS lines, such as #PBS -N, which sets the name of the job.

The rest of the script is run as a bash script at run time. A bash shell on the first node of the two nodes that are requested executes these commands as a normal bash script, just as if you had run this as a shell script from the terminal. The only difference is that PBS sets certain environment variables that you can use in the script. $PBS_O_WORKDIR is set to be the directory that the command was 'submitted' from - eg, $SCRATCH/SOMEDIRECTORY. The script then uses the mpirun command to launch the job.

Submitting Collections of Serial Jobs

You cannot run purely serial jobs on the GPC (or any of SciNet's systems), as this would mean only one core out of 8 is used. If you have serial jobs, you have to bunch them together. SciNet-approved methods for running collections of serial jobs can be found on the serial run wiki page.

Batch Submission Script: OpenMP

For running OpenMP jobs, the procedure is similar as for MPI jobs:

<source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (OpenMP)
4. PBS -l nodes=1:ppn=8,walltime=1:00:00
5. PBS -N test

1. load modules (must match modules used for compilation)

module load intel

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=8 ./a.out </source>

Note that in some circumstances it can be more efficient to run (say) two jobs each running on four threads than one job running on eight threads. In that case you can use the same `ampersand-and-wait' technique outlined for serial jobs (see serial run wiki page) for less-than-eight-core OpenMP jobs.

Hybrid MPI/OpenMP jobs

Using Intel MPI

Here is how to run hybrid codes using intelmpi::

http://software.intel.com/en-us/articles/hybrid-applications-intelmpi-openmp/

Make sure you compile with the -mt_mpi option to the compilers to use the thread safe libraries. Set the environment variable I_MPI_PIN_DOMAIN:

$ export I_MPI_PIN_DOMAIN=omp

This will set the process pinning domain size to be equal to OMP_NUM_THREADS (which you should set to the desired number of threads per mpi process). Therefore, each MPI process can create $OMP_NUM_THREADS number of children threads for running within the corresponding domain. If OMP_NUM_THREADS is not set, each node is treated as a separate domain (which will allow as many threads per MPI processes as there are cores).

In addition, when invoking mpirun, you should add the argument "-ppn X", where X is the number of MPI processes per node. For example:

$ mpirun -ppn 2 -np 8 [executable]

would start 2 mpi processes of [executable] per node for a total of 8 processes, so mpirun will try to run mpi processes on 4 nodes (OMP_NUM_THREADS is then probably best set at 4). Your job script should still ask for these 4 nodes with the line <source lang="bash">

    #PBS -l nodes=4:ppn=8,walltime=....

</source> (ppn=8 is not a mistake here; the ppn parameter has a different meaning for PBS and for mpirun)

The ppn parameter to mpirun is very important! Without it, eight mpi jobs would get bunched on the first node in this example, leaving 3 nodes unused.

NOTE: In order to pin OpenMP threads inside the domain, use the corresponding OpenMP feature by setting the KMP_AFFINITY environment variable, see Compiler User and Reference Guide.

The IntelMPI manual is referenced on the front page of our wiki:

http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf

For the above example of a total of 8 processes on 4 nodes, you could use the following script: <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (hybrid job)
4. PBS -l nodes=4:ppn=8,walltime=1:00:00
5. PBS -N test

1. load modules (must match modules used for compilation)

module load intel intelmpi

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. SET THE NUMBER OF THREADS PER PROCESS:

export OMP_NUM_THREADS=4

1. PIN THE MPI DOMAINS ACCORDING TO OMP

export I_MPI_PIN_DOMAIN=omp

1. EXECUTION COMMAND; -np = nodes*ppn

mpirun -ppn 2 -np 8 ./a.out </source>

Using Open MPI

For mixed MPI/OpenMP jobs using OpenMPI, which is the default for many users, the procedure is similar, but details differ.

• Request the number of nodes in the PBS script.
• Set OMP_NUM_THREADS to the number of threads per MPI process.
• In addition to the -np parameter for mpirun, add the argument --bynode, so that the mpi processes are not bunched up.

So for example, to start a total of 8 processes on 4 nodes, you could use the following script <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (hybrid job)
4. PBS -l nodes=4:ppn=8,walltime=1:00:00
5. PBS -N test

1. load modules (must match modules used for compilation)

module load intel openmpi

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. SET THE NUMBER OF THREADS PER PROCESS:

export OMP_NUM_THREADS=4

1. EXECUTION COMMAND; -np = nodes*processes_per_nodes; --byhost forces a round robin of nodes.

mpirun -np 8 --bynode ./a.out </source>

• More questions about running your jobs? See the FAQs Running your jobs and Errors in running jobs sections.

Submitting an Interactive (Debug) Job

Your development work flow may require a lot of small test runs. You are allowed to do these on the development nodes, as long as it's very brief (a few minutes), and does not use all cores on the machine. For anything more you will have to use the compute nodes.

It is sometimes convenient to run a job interactively; this can be very handy for debugging purposes. In this case, you type a qsub command which submits an interactive job to the queue; when the scheduler selects this job to run, then it starts a shell running on the first node of the job, which connects to your terminal. You can then type any series of commands (for instance, the same commands listed as in the batch submission script above) to run a job interactively.

For example, to start the same sort of job as in the batch submission script above, but interactively, one would type

$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00

This is exactly the #PBS -l line in the batch script above (which requests all 8 processors on each of 2 nodes for one hour), but prepended with a -I for `interactive'. When this job begins, your terminal will now show you as being logged in to one of the compute nodes, and one can type in any shell command, run mpirun, etc. When you exit the shell, the job will end. Interactive jobs can be used with any of the GPC queues however, there is a short high turnover queue called debug which can be especially useful when the system is busy.

• More questions about running test jobs? See the FAQ.

QDR vs. DDR Infiniband

The GPC Infiniband network GPC has two sections, one connecting 3,024 nodes with 5:1 blocking/oversubscribed QDR and the second connecting 840 nodes 1:1 non-blocking DDR Infiniband. By default a user's job will go to whichever network section best accommodates it, typically smaller jobs to the QDR and larger jobs to the DDR. However a user can override this by simply adding the flags "ddr" or "qdr" to the job resource request.

For example, to request two nodes anywhere on the GPC (QDR or DDR), use

#PBS -l nodes=2:ppn=8,walltime=1:00:00

in your job submission script.

For two nodes using DDR, use

#PBS -l nodes=2:ddr:ppn=8,walltime=1:00:00

To get two nodes using QDR, instead, you would use

#PBS -l nodes=2:qdr:ppn=8,walltime=1:00:00

The queueing system also tries its best to keep jobs within the same switch of the QDR thus avoiding the 5:1 blocking. A user can can explicitly request this behaviour if their jobs is less than 30 nodes using

#PBS -W x=nodesetisoptional:false

HyperThreading

Each GPC compute node has 8 Nehalem cores (2 sockets each with a four-core Intel Xeon E5540 @ 2.53GHz). Thus, to make full use of the computing power of a GPC node, you must be running least 8 "tasks" -- MPI processes, or OpenMP threads.

Under most circumstances, running exactly 8 tasks is the most efficient way to use these nodes. However, sometimes software design (eg, having one thread for communication and one for computation) can usefully `oversubscribe' the number of physical cores, and running (say) twice as many tasks as cores can be a useful strategy. If your code is highly memory-bandwidth bound, having one task ready to run while another waits for memory access can make more effective use of the processor.

The Nehalem processors have hardware support for such two-way overloading of processors, through "HyperThreading"; there are an extra set of registers on each core to facilitate rapid switching between two tasks, making it look to the operating system that there are in fact 16 cores per node. Depending on the nature of your code, making use of these virtual extra cores may speed up or slow down your computation; you should run small test cases before running production jobs in this manner. In most cases, the speed difference will be under 10%. Some of our users have obtained an 8% speedup by running gromacs with 16 tasks instead of 8 on a single node (mpirun -np 16 ./gromacs/mdrun -npme 4 is 108% the speed of mpirun -np 8 ./gromacs/mdrun with -npme 2 or -1).

HyperThreading with OpenMP

To use hyperthreading with an OpenMP job, one just runs twice as many threads as one would have previously; eg, if you were running 8 threads before (export OMP_NUM_THREADS=8) you would run with 16 (export OMP_NUM_THREADS=16). Everything else remains the same, including the job submission script; one still uses ppn=8 in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

HyperThreading with MPI

To use hyperthreading with an MPI job, one just runs twice as many MPI processes as one would have previously; eg, if you were running on three nodes using 8 MPI tasks per node and used mpirun ... -np 24, you could run instead with -np 48. Everything else remains the same, including the job submission script; one still uses ppn=8 in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

Note that if you are using OpenMPI (as is the default), there is another consideration; OpenMPI assumes that there is no oversubscription and each task very aggressively makes full use of a core when it is waiting for a message (eg, the waits are "busywaits"). If you find a significant slowdown when running multiple MPI tasks per core with OpenMPI, you may want to try adding the additional option to mpirun: --mca mpi_yield_when_idle 1. This will increase the latency of individual messages, but free up the core to do additional work while waiting.

With IntelMPI, the problem should be less pronounced, but you can still improve things by using mpirun -genv I_MPI_SPIN_COUNT 1 ...

Examples of hyperthreading with MPI

Hyperthreading using gromacs: https://support.scinet.utoronto.ca/wiki/index.php/Gromacs#Hyperthreading_with_Gromacs

HyperThreading with Hybrid MPI/OpenMP codes

With a hybrid code, one has extra flexibility in how to assign the "extra" cores -- you could run extra MPI tasks or extra OpenMPI threads. As with all hybrid codes, the combination which results in the best performance depends very strongly on the nature of your code, and you should experiment with different combinations. In addition, with hybrid codes processor and memory affinity issues become very important; if you're unsure as to how to tune your application for best performance, please make an appointment with the SciNet technical analysts for more help.

Memory Configuration

16G

There are 3756 nodes which have 16G of memory, and is the primary configuration in the GPC. These nodes will be used by default. On these nodes, about 2 GB is taken by the operating system. So for mpi runs with 8 processes per node, this leaves about 1.75GB max per mpi process. Do not try to use more than the available memory: the node will crash and your job will either die or hang until the requested walltime has elapsed.

If you need more memory per process or per thread, you can either try to use the limit number of larger memory nodes listed below, or you can run with less mpi processes, or use a different decomposition, such that the job fits on a node.

32G

There are 84 nodes which have 32G of memory. To request these nodes use:

$ qsub -l nodes=2:m32g:ppn=8,walltime=1:00:00 

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total of 32GB.

64G/128G/256G

There are 72 16-core Intel Sandybridge nodes which have 64G, 2 with 128G, and 2 with 256G of memory available as part of the contributed Sandy cluster. These nodes are requested through the sandy queue.

$ qsub -l nodes=1:m64g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m256g:ppn=16,walltime=1:00:00 -q sandy

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total.

128G

There are two other stand-alone large memory (128GB) nodes which are primarily to be used for data analysis of runs. They have 16 cores and are intel machines running linux, but they are of a different architecture than the GPC compute nodes, so codes may have to be compiled separately for these machines. This also means that some modules that work on the other GPC nodes, such as octave, will not work in the 128G nodes.

These nodes can be accessed using a specific largemem queue.

$ qsub -l nodes=2:ppn=16,walltime=1:00:00 -q largemem -I

Note: To estimate your time of access to these nodes, use

$ showq -w class=largemem

Ram Disk

On the GPC nodes, there is a `ram disk' available - up to half of the memory on the node may be used as a temporary file system. This is particularly useful for use in the early stages of migrating destop-computing codes to a High Performance Computing platform such as the GPC. It is much faster than real disk and does not require network traffic; however, each node sees its own ramdisk and cannot see files on that of other nodes. This is a very easy way to cache writes (by writing them to fast ram disk instead of slow `real' disk); and then one would periodically copy the files to files on /scratch or /project so that they are available after the job has completed.

To use the ramdisk, create and read to or write from files in /dev/shm/... just as one would to (eg) $SCRATCH. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

NOTE: it is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

More details on how to setup your script to use the ramdisk can be found on the Ramdisk wiki page.

Managing jobs on the Queuing system

Information on checking available resources, starting, viewing, managing and canceling jobs on Moab/Torque. Also check out the FAQ.