GPC Quickstart

General Purpose Cluster (GPC)
Installed	June 2009
Operating System	Linux
Interconnect	1/4 on Infiniband, rest on GigE
Ram/Node	16 Gb
Cores/Node	8
Login/Devel Node	gpc01..gpc04 (from login.scinet)
Vendor Compilers	icc (C) ifort (fortran) icpc (C++)
Queue Submission	Moab/Torque

Specifications

The General Purpose Cluster is an extremely large cluster (ranked 16th in the world at its inception, and fastest in Canada) and is where most computations are to be done at SciNet. It is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the first in the world to make use of the new chips). The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,240 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking 4x-DDR InfiniBand while the rest of the nodes are connected with gigabit ethernet. The compute nodes are accessed through a queuing system that allows jobs with a maximum wall time of 48 hours.

Login

First login via ssh with your SciNet account at login.scinet.utoronto.ca, and from there you can proceed to the Development nodes (gpc01,gpc02,gpc03, or gpc04) to compile/test your code.

Compile/Devel Nodes

From a scinet login node you can ssh to gpc01..gpc04. You may also just use /scinet/gpc/bin/gpcdev to take directly to the dev node with lowest cpu load (reassessed every 5 minutes).

% /scinet/gpc/bin/gpcdev -h
  Usage: gpcdev [usual ssh options]
Example: gpcdev -X
redirected to dev node with lowest cpu load

These nodes have the same hardware configuration as most of the compute nodes -- 8 Nehalem processing cores with 16GB RAM and Gigabit ethernet. You can compile and test your codes on these nodes. To interactively test on more than 8 processors, or to test your code over an InfiniBand connection, you can submit an interactive job request.

Your home directory is in /home/USER; you have 10GB there that is backed up. Your home directory cannot be written to by the compute nodes! Thus, to run jobs, you'll use the /scratch/USER directory. Here, there is a large amount of disk space, but it is not backed up. Thus it makes sense to keep your codes in /home, compile there, and then run them in the /scratch directory.

Modules and Environment Variables

To use most packages on the SciNet machines - including any of the compilers - , you will have to use the `modules' command. The command module load some-package will set your environment variables (PATH, LD_LIBRARY_PATH, etc) to include the default version of that package. module load some-package/specific-version will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

Note that to use even the gcc compilers you will have to do

$ module load gcc

but in fact you probably should use the intel compilers installed on this system as they usually produce faster code (and sometimes, much faster.)

A list of the installed software is available in Software & Libraries and can be seen on the system by typing

$ module avail

To load a module (for example, the default version of the intel compilers)

$ module load intel

To unload a module

$ module unload intel

To unload all modules

$ module purge

These commands should go in your .bashrc files and/or in your submission scripts to make sure you are using the correct packages.

Compilers

The intel compilers are icc/icpc/ifort for C/C++/Fortran, and are available with the default module "intel". The intel compilers are recommended over the GNU compilers. Documentation about icpc is available at http://software.intel.com/en-us/articles/intel-software-technical-documentation/. The Intel compilers accept many of the options that the GNU compilers accept, but tend to produce faster programs on our system. If, for some reason, you really need the GNU compilers, the latest version of the GNU compiler collection (currently 4.4.0) is available by loading the "gcc" module, with gcc/g++/gfortran for C/C++/Fortran. Note that f77/g77 is not supported.

To ensure that the intel compilers are in your PATH and their libraries are in your LD_LIBRARY_PATH, use the command

$ module load intel

This should likely go in your .bashrc file so that it will automatically be loaded.

Optimize your code for the GPC machine using of at least the following compiler flags:

   -O3 -xHost

(or -O3 -march=native for the GNU compilers).

• If your program uses openmp, add -openmp (-fopenmp for GNU compilers).
• If you get the warning feupdatreenv is not implemented, add -limf to the link line.
• If you need to link in the MKL libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code. Note that this give the link line for the command prompt. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.
• More questions about compiling? See the FAQ.

Debuggers

• ddt - Allinea's graphical parallel debugger, in the ddt module. Highly recommended!
• gdb - The GNU Debugger, available in the gdb module.
• idbc/idb - The intel debuggers, part of the intel module(s).
• ddd - A graphical debuggerThe GNU Debugger, available in the ddd module.

Note that to debug code, you have to give the

-g

flags to the compiler. The intel compiler needs the additional option -debug parallel to debug threaded/OpenMP code.

MPI

SciNet currently provides multiple MPI libraries for the GPC; OpenMPI, and IntelMPI. We currently recommend OpenMPI as the default, as it quite reliably demonstrates good performance on both the infiniband and ethernet networks. For full details and options see the complete MPI section.

The MPI libraries are compiled with both the gnu compiler suite and the intel compiler suite. To use (for instance) the intel-compiled OpenMPI libraries, which we recommend as the default (and use for most of our examples here), use

$ module load openmpi intel

in your .bashrc. Other combinations behave similarly.

The MPI libraries define the wrappers mpicc/mpicxx/mpif90/mpif77 as wrappers around the appropriate compilers, which ensure the appropriate include and library directories and used in the compilation and linking steps.

We currently recommend the Intel + OpenMPI combination. However, if you require the GNU compilers as well as MPI, you would want to find the most recent openmpi module available with `gcc' in the version name. This will enable development and runtime with gcc/g++/gfortran and OpenMPI. You can make this your default by putting the module load line in your ~/.bashrc file.

For mixed OpenMP/MPI code using Intel MPI, add the compilation flag -mt_mpi for full thread-safety (no such flag is necessary for OpenMPI).

Submitting A Batch Job

The SciNet machines are shared systems, and jobs that are to run on them are submitted to a queue; the scheduler then orders the jobs in order to make the best use of the machine, and has them launched when resources become availble. The intervention of the scheduler can mean that the jobs aren't quite run in a first-in first-out order.

The scheduler will have the job run on one or more of the compute nodes of the GPC. It is important to realize that on compute nodes, your home directory is read-only. You have to run your jobs from the /scratch/$USER directory instead. See Data Management for more details on the file systems at SciNet.

The maximum wallclock time for a job in the queue is 48 hours; computations that will take longer than this must be broken into 48-hour chunks and run as several jobs. The usual way to do this is with checkpoints, writing out the complete state of the computation every so often in such a way that a job can be restarted from this state information and continue on from where it left off. Generating checkpoints is a good idea anyway, as in the unlikely event of a hardware failure during your run, it allows you to restart without having lost much work.

There are limits to how many jobs you can submit. If your group has a default account, up to 32 nodes at a time for 48 hours per job on the GPC cluster are allowed to be queued. This is a total limit, e.g., you could request 64 nodes for 24 hours. Jobs of users with an LRAC or NRAC allocation will run at a higher priority than others while their resources last. Because of the group-based allocation, it is conceivable that your jobs won't run if your colleagues have already exhausted your group's limits.

Note that scheduling big jobs greatly affects the queuer and other users, so you have to talk to us first to run massively parallel jobs (> 2048 cores). We will help make sure that your jobs start and run efficiently.

If your job should run in fewer than 48 hours, specify that in your script -- your job will start sooner. (It's easier for the scheduler to fit in a short job than a long job). On the downside, the job will be killed automatically by the queue manager software at the end of the specified wallclock time, so if you guess wrong you might lose some work. So the standard procedure is to estimate how long your job will take and add 10% or so.

You interact with the queuing system through the queue/resource manager, Moab and Torque. To see all the jobs in the queue use

$ showq

To submit your own job, you must write a script which describes the job and how it is to be run (a sample script follows) and submit it to the queue, using the command

$ qsub [SCRIPT-FILE-NAME]

where you will replace [SCRIPT-FILE-NAME] with the file containing the submission script. This will return a job ID, for example 31415, which is used to identify the jobs. Information about a queued job can be found using

$ checkjob [JOB-ID]

and jobs can be canceled with the command

$ canceljob [JOB-ID]

Again, these commands have many options, which can be read about on their man pages.

Much more information on the queueing system is available on our queue page.

Batch Submission Script: MPI

A sample submission script is shown below for an mpi job using ethernet with the #PBS directives at the top and the rest being what will be executed on the compute node.

<source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (ethernet)
4. PBS -l nodes=2:ppn=8,walltime=1:00:00
5. PBS -N test

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. EXECUTION COMMAND; -np = nodes*ppn

mpirun -np 16 -hostfile $PBS_NODEFILE ./a.out </source>

The lines that begin #PBS are commands that are parsed and interpreted by qsub at submission time, and control administrative things about your job. In this example, the script above requests two nodes, using 8 processors per node, for a wallclock time of one hour. (The resources required by the job are listed on the #PBS -l line.) Other options can be given in other #PBS lines, such as #PBS -N, which sets the name of the job.

The rest of the script is run as a bash script at run time. A bash shell on the first node of the two nodes that are requested executes these commands as a normal bash script, just as if you had run this as a shell script from the terminal. The only difference is that PBS sets certain environment variables that you can use in the script. $PBS_O_WORKDIR is set to be the directory that the command was 'submitted' from - eg, /scratch/USER/SOMEDIRECTORY - and $PBS_NODEFILE is the name of a file which contains all the nodes on which programs should execute. Using these environment variables, the script then uses the mpirun command to launch the job. Assumed here is that the user has a line like

$ module load openmpi intel

in their .bashrc.

• Note: The different versions of MPI require different commands to launch the run, and thus different scripts. The above script is specific for the openmpi module. For the intelmpi module on ethernet, the last line of the script should read

$ mpirun -np 16 -env I_MPI_FABRICS shm:tcp ./a.out

For full MPI details see MPI

Submitting Collections of Serial Jobs

SciNet-approved methods for running collections of serial jobs can be found on the serial run wiki page.

Batch Submission Script: OpenMP

For running OpenMP jobs, the procedure is similar as for MPI jobs:

<source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (OpenMP)
4. PBS -l nodes=1:ppn=8,walltime=1:00:00
5. PBS -N test

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=8 ./a.out </source>

Note that in some circumstances it can be more efficient to run (say) two jobs each running on four threads than one job running on eight threads. In that case you can use the same `ampersand-and-wait' technique outlined for serial jobs (see serial run wiki page) for less-than-eight-core OpenMP jobs.

Hybrid MPI/OpenMP jobs

Using Intel MPI

Here is how to run hybrid codes using intelmpi::

http://software.intel.com/en-us/articles/hybrid-applications-intelmpi-openmp/

Make sure you compile with the -mt_mpi option to the compilers to use the thread safe libraries. Set the environment variable I_MPI_PIN_DOMAIN:

$ export I_MPI_PIN_DOMAIN=omp

This will set the process pinning domain size to be equal to OMP_NUM_THREADS (which you should set to the desired number of threads per mpi process). Therefore, each MPI process can create $OMP_NUM_THREADS number of children threads for running within the corresponding domain. If OMP_NUM_THREADS is not set, each node is treated as a separate domain (which will allow as many threads per MPI processes as there are cores).

In addition, when invoking mpirun, you should add the argument "-ppn X", where X is the number of MPI processes per node. For example:

$ mpirun -r ssh -ppn 2 -np 8 [executable]

would start 2 mpi processes of [executable] per node for a total of 8 processes, so mpirun will try to run mpi processes on 4 nodes (OMP_NUM_THREADS is then probably best set at 4). Your job script should still ask for these 4 nodes with the line <source lang="bash">

    #PBS -l nodes=4:ppn=8,walltime=....

</source> (ppn=8 is not a mistake here; the ppn parameter has a different meaning for PBS and for mpirun)

The ppn parameter to mpirun is very important! Without it, eight mpi jobs would get bunched on the first node in this example, leaving 3 nodes unused.

NOTE: In order to pin OpenMP threads inside the domain, use the corresponding OpenMP feature by setting the KMP_AFFINITY environment variable, see Compiler User and Reference Guide.

The IntelMPI manual is referenced on the front page of our wiki:

http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf

For the above example of a total of 8 processes on 4 nodes, you could use the following script: <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (hybrid job)
4. PBS -l nodes=4:ppn=8,walltime=1:00:00
5. PBS -N test

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. SET THE NUMBER OF THREADS PER PROCESS:

export OMP_NUM_THREADS=4

1. PIN THE MPI DOMAINS ACCORDING TO OMP

export I_MPI_PIN_DOMAIN=omp

1. EXECUTION COMMAND; -np = nodes*ppn

mpirun -r ssh -ppn 2 -np 8 ./a.out </source>

Using Open MPI

For mixed MPI/OpenMP jobs using OpenMPI, which is the default for many users, the procedure is similar, but details differ.

• Request the number of nodes in the PBS script.
• Set OMP_NUM_THREADS to the number of threads per MPI process.
• In addition to the -np parameter for mpirun, add the argument --bynode, so that the mpi processes are not bunched up.

So for example, to start a total of 8 processes on 4 nodes, you could use the following script <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for SciNet GPC (hybrid job)
4. PBS -l nodes=4:ppn=8,walltime=1:00:00
5. PBS -N test

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. SET THE NUMBER OF THREADS PER PROCESS:

export OMP_NUM_THREADS=4

1. EXECUTION COMMAND; -np = nodes*processes_per_nodes; --byhost forces a round robin of nodes.

mpirun -np 8 --bynode -hostfile $PBS_NODEFILE ./a.out </source>

• More questions about running your jobs? See the FAQs Running your jobs and Errors in running jobs sections.

Submitting an Interactive (Debug) Job

Your development work flow may require a lot of small test runs. You are allowed to do these on the development nodes, as long as it's very brief (a few minutes), and does not use all cores on the machine. For anything more you will have to use the compute nodes.

It is sometimes convenient to run a job interactively; this can be very handy for debugging purposes. In this case, you type a qsub command which submits an interactive job to the queue; when the scheduler selects this job to run, then it starts a shell running on the first node of the job, which connects to your terminal. You can then type any series of commands (for instance, the same commands listed as in the batch submission script above) to run a job interactively.

For example, to start the same sort of job as in the batch submission script above, but interactively, one would type

$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00

This is exactly the #PBS -l line in the batch script above (which requests all 8 processors on each of 2 nodes for one hour), but prepended with a -I for `interactive'. When this job begins, your terminal will now show you as being logged in to one of the compute nodes, and one can type in any shell command, run mpirun, etc. When you exit the shell, the job will end. Interactive jobs can be used with any of the GPC queues however, there is a short high turnover queue called debug which can be especially useful when the system is busy.

• More questions about running test jobs? See the FAQ.

Ethernet vs. Infiniband

About 1/4 of the GPC (862 nodes or 6896 cores) is connected with a high bandwidth low-latency fabric called InfiniBand. Many jobs which require tight coupling to scale well greatly benefit from this interconnect; other types of jobs, which have relatively modest communications, do not require this and run fine on Gigabit ethernet.

Jobs which require the InfiniBand for good performance can request the nodes that have the `ib' feature in the #PBS -l line, <source lang="bash">

1. PBS -l nodes=2:ib:ppn=8,walltime=1:00:00

</source> Furthermore, if your mpirun command specifically requests a fabric in its options (eg. ssm), you will have to change those options as well. See GPC MPI Versions.

Because there are a limited number of these nodes, your job will start running faster if you do not request them (e.g. if you use the scripts as shown above), as this increases the number of nodes available to run your job. In fact, the InfiniBand nodes are to be used only for jobs that are known to scale well and will benefit from this type of interconnect. As such the minimum number of nodes requested has to be at least 2, as single node jobs will not benefit from using an Infiniband node. The MPI libraries provided by SciNet automatically correctly use either the InfiniBand or ethernet interconnect depending on which nodes your job runs on.

HyperThreading

Each GPC compute node has 8 Nehalem cores (2 sockets each with a four-core Intel Xeon E5540 @ 2.53GHz). Thus, to make full use of the computing power of a GPC node, you must be running least 8 "tasks" -- MPI processes, or OpenMP threads.

Under most circumstances, running exactly 8 tasks is the most efficient way to use these nodes. However, sometimes software design (eg, having one thread for communication and one for computation) can usefully `oversubscribe' the number of physical cores, and running (say) twice as many tasks as cores can be a useful strategy. If your code is highly memory-bandwidth bound, having one task ready to run while another waits for memory access can make more effective use of the processor.

The Nehalem processors have hardware support for such two-way overloading of processors, through "HyperThreading"; there are an extra set of registers on each core to facilitate rapid switching between two tasks, making it look to the operating system that there are in fact 16 cores per node. Depending on the nature of your code, making use of these virtual extra cores may speed up or slow down your computation; you should run small test cases before running production jobs in this manner. In most cases, the speed difference will be under 10%. Some of our users have obtained an 8% speedup by running gromacs with 16 tasks instead of 8 on a single node (mpirun -np 16 ./gromacs/mdrun -npme 4 is 108% the speed of mpirun -np 8 ./gromacs/mdrun with -npme 2 or -1).

HyperThreading with OpenMP

To use hyperthreading with an OpenMP job, one just runs twice as many threads as one would have previously; eg, if you were running 8 threads before (export OMP_NUM_THREADS=8) you would run with 16 (export OMP_NUM_THREADS=16). Everything else remains the same, including the job submission script; one still uses ppn=8 in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

HyperThreading with MPI

To use hyperthreading with an MPI job, one just runs twice as many MPI processes as one would have previously; eg, if you were running on three nodes using 8 MPI tasks per node and used mpirun ... -np 24, you could run instead with -np 48. Everything else remains the same, including the job submission script; one still uses ppn=8 in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

Note that if you are using OpenMPI (as is the default), there is another consideration; OpenMPI assumes that there is no oversubscription and each task very aggressively makes full use of a core when it is waiting for a message (eg, the waits are "busywaits"). If you find a significant slowdown when running multiple MPI tasks per core with OpenMPI, you may want to try adding the additional option to mpirun: --mca mpi_yield_when_idle 1. This will increase the latency of individual messages, but free up the core to do additional work while waiting.

With IntelMPI, the problem should be less pronounced, but you can still improve things by using mpirun -genv I_MPI_SPIN_COUNT 1 ...

Examples of hyperthreading with MPI

Hyperthreading using gromacs: https://support.scinet.utoronto.ca/wiki/index.php/Gromacs#Hyperthreading_with_Gromacs

HyperThreading with Hybrid MPI/OpenMP codes

With a hybrid code, one has extra flexibility in how to assign the "extra" cores -- you could run extra MPI tasks or extra OpenMPI threads. As with all hybrid codes, the combination which results in the best performance depends very strongly on the nature of your code, and you should experiment with different combinations. In addition, with hybrid codes processor and memory affinity issues become very important; if you're unsure as to how to tune your application for best performance, please make an appointment with the SciNet technical analysts for more help.

Memory Configuration

16G

There are 3756 nodes which have 16G of memory, and is the primary configuration in the GPC. These nodes will be used by default. On these nodes, about 2 GB is taken by the operating system. So for mpi runs with 8 processes per node, this leaves about 1.75GB max per mpi process. Do not try to use more than the available memory: the node will crash and your job will either die or hang until the requested walltime has elapsed.

If you need more memory per process or per thread, you can either try to use the limit number of larger memory nodes listed below, or you can run with less mpi processes, or use a different decomposition, such that the job fits on a node.

18G

There are 24 Infiniband nodes which have 18G of memory. These nodes have a fully populated memory configuration that maximizes memory bandwidth. Note that also on these nodes, about 2 GB is taken by the operating system. To request these nodes use:

$ qsub -l nodes=2:ib:m18g:ppn=8,walltime=1:00:00 

32G

There are 84 Infiniband nodes which have 32G of memory. To request these nodes use:

$ qsub -l nodes=2:ib:m32g:ppn=8,walltime=1:00:00 

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total of 32GB.

128G

There are two stand-alone large memory (128GB) nodes, gpc-lrgmem01 and gpc-lrgmem02 which are primarily to be used for data analysis of runs. They have 16 cores and are intel machines running linux, but they are of a different architecture than the GPC compute nodes, so codes may have to be compiled separately for these machines. This also means that some modules that work on the other GPC nodes, such as octave, will not work in the 128G nodes.

These nodes can be accessed using a specific largemem queue.

$ qsub -l nodes=2:ppn=8,walltime=1:00:00 -q largemem -I

Note: To estimate your time of access to these nodes, use

$ showq -w class=largemem

because showstart seems to always return "INFINITY".

Ram Disk

On the GPC nodes, there is a `ram disk' available - up to half of the memory on the node may be used as a temporary file system. This is particularly useful for use in the early stages of migrating destop-computing codes to a High Performance Computing platform such as the GPC. It is much faster than real disk and does not require network traffic; however, each node sees its own ramdisk and cannot see files on that of other nodes. This is a very easy way to cache writes (by writing them to fast ram disk instead of slow `real' disk); and then one would periodically copy the files to files on /scratch or /project so that they are available after the job has completed.

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) /scratch/USER/. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

NOTE: it is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

More details on how to setup your script to use the ramdisk can be found on the Ramdisk wiki page.

Managing jobs on the Queuing system

Information on checking available resources, starting, viewing, managing and canceling jobs on Moab/Torque. Also check out the FAQ.