GPC Quickstart
| General Purpose Cluster (GPC) | |
|---|---|
| Installed | June 2009 |
| Operating System | Linux |
| Number of Nodes | 3864 (30,912 cores) |
| Interconnect | 840 nodes 1:1 DDR, 3024 nodes 5:1 QDR |
| Ram/Node | 16 Gb |
| Cores/Node | 8 (16 threads) |
| Login/Devel Node | gpc01..gpc04 (from login.scinet) |
| Vendor Compilers | icc (C) ifort (fortran) icpc (C++) |
| Queue Submission | Moab/Torque |
Specifications
The General Purpose Cluster is an extremely large cluster (ranked 16th in the world at its inception, and fastest in Canada) and is where most computations are to be done at SciNet. It is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the first in the world to make use of the new chips). The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,912 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking DDR InfiniBand while the rest of the nodes are connected with 5:1 blocked QDR InfiniBand. The compute nodes are accessed through a queuing system that allows jobs with a maximum wall time of 48 hours.
Login
First login via ssh with your SciNet account at login.scinet.utoronto.ca, and from there you can proceed to the Development nodes (gpc01,gpc02,gpc03, or gpc04) to compile/test your code.
Compile/Devel Nodes
From a scinet login node you can ssh to gpc01..gpc04. You may also just use /scinet/gpc/bin/gpcdev to take directly to the dev node with lowest cpu load (reassessed every 5 minutes).
% /scinet/gpc/bin/gpcdev -h Usage: gpcdev [usual ssh options] Example: gpcdev -X redirected to dev node with lowest cpu load
These nodes have the same hardware configuration as most of the compute nodes -- 8 Nehalem processing cores with 16GB RAM and Gigabit ethernet. You can compile and test your codes on these nodes. To interactively test on more than 8 processors, or to test your code over an InfiniBand connection, you can submit an interactive job request.
Your home directory is in $HOME (currently /home/USER but soon to move to /home/g/group/USER - so make sure you use the environment variable $HOME or ~ in scripts). You have 10GB there that is backed up. Your home directory cannot be written to by the compute nodes! Thus, to run jobs, you'll use the $SCRATCH directory (currently /scratch/USER but soon to move to /scratch/g/group/USER - again, use the environment variable). Here, there is a large amount of disk space, but it is not backed up. Thus it makes sense to keep your codes in /home, compile there, and then run them in the /scratch directory.
Modules and Environment Variables
To use most packages on the SciNet machines - including any of the compilers - , you will have to use the `modules' command. The command module load some-package will set your environment variables (PATH, LD_LIBRARY_PATH, etc) to include the default version of that package. module load some-package/specific-version will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.
Note that to use even the gcc compilers you will have to do
$ module load gcc
but in fact you probably should use the intel compilers installed on this system as they usually produce faster code (and sometimes, much faster.)
A list of the installed software is available in Software & Libraries and can be seen on the system by typing
$ module avail
To load a module (for example, the default version of the intel compilers)
$ module load intel
To unload a module
$ module unload intel
To unload all modules
$ module purge
These commands should go in your .bashrc files and/or in your submission scripts to make sure you are using the correct packages.
Compilers
The intel compilers are icc/icpc/ifort for C/C++/Fortran, and are available with the default module "intel". The intel compilers are recommended over the GNU compilers. Documentation about icpc is available at http://software.intel.com/en-us/articles/intel-software-technical-documentation/. The Intel compilers accept many of the options that the GNU compilers accept, but tend to produce faster programs on our system. If, for some reason, you really need the GNU compilers, the latest version of the GNU compiler collection (currently 4.4.0) is available by loading the "gcc" module, with gcc/g++/gfortran for C/C++/Fortran. Note that f77/g77 is not supported.
To ensure that the intel compilers are in your PATH and their libraries are in your LD_LIBRARY_PATH, use the command
$ module load intel
This should likely go in your .bashrc file so that it will automatically be loaded.
Optimize your code for the GPC machine using of at least the following compiler flags:
-O3 -xHost
(or -O3 -march=native for the GNU compilers).
- If your program uses openmp, add -openmp (-fopenmp for GNU compilers).
- If you get the warning feupdatreenv is not implemented, add -limf to the link line.
- If you need to link in the MKL libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code. Note that this give the link line for the command prompt. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.
- More questions about compiling? See the FAQ.
Debuggers
- ddt - Allinea's graphical parallel debugger, in the ddt module. Highly recommended!
- gdb - The GNU Debugger, available in the gdb module.
- idbc/idb - The intel debuggers, part of the intel module(s).
- ddd - A graphical debuggerThe GNU Debugger, available in the ddd module.
Note that to debug code, you have to give the
-g
flags to the compiler. The intel compiler needs the additional option -debug parallel to debug threaded/OpenMP code.
MPI
SciNet currently provides multiple MPI libraries for the GPC; OpenMPI, and IntelMPI. We currently recommend OpenMPI as the default, as it quite reliably demonstrates good performance on both the infiniband and ethernet networks. For full details and options see the complete MPI section.
The MPI libraries are compiled with both the gnu compiler suite and the intel compiler suite. To use (for instance) the intel-compiled OpenMPI libraries, which we recommend as the default (and use for most of our examples here), use
$ module load openmpi intel
in your .bashrc. Other combinations behave similarly.
The MPI libraries define the wrappers mpicc/mpicxx/mpif90/mpif77 as wrappers around the appropriate compilers, which ensure the appropriate include and library directories and used in the compilation and linking steps.
We currently recommend the Intel + OpenMPI combination. However, if you require the GNU compilers as well as MPI, you would want to find the most recent openmpi module available with `gcc' in the version name. This will enable development and runtime with gcc/g++/gfortran and OpenMPI. You can make this your default by putting the module load line in your ~/.bashrc file.
For mixed OpenMP/MPI code using Intel MPI, add the compilation flag -mt_mpi for full thread-safety (no such flag is necessary for OpenMPI).
Submitting A Batch Job
The SciNet machines are shared systems, and jobs that are to run on them are submitted to a queue; the scheduler then orders the jobs in order to make the best use of the machine, and has them launched when resources become availble. The intervention of the scheduler can mean that the jobs aren't quite run in a first-in first-out order.
The scheduler will have the job run on one or more of the compute nodes of the GPC. It is important to realize that on compute nodes, your home directory is read-only. You have to run your jobs from the $SCRATCH directory instead. See Data Management for more details on the file systems at SciNet.
The maximum wallclock time for a job in the queue is 48 hours; computations that will take longer than this must be broken into 48-hour chunks and run as several jobs. The usual way to do this is with checkpoints, writing out the complete state of the computation every so often in such a way that a job can be restarted from this state information and continue on from where it left off. Generating checkpoints is a good idea anyway, as in the unlikely event of a hardware failure during your run, it allows you to restart without having lost much work.
There are limits to how many jobs you can submit. If your group has a default account, up to 32 nodes at a time for 48 hours per job on the GPC cluster are allowed to be queued. This is a total limit, e.g., you could request 64 nodes for 24 hours. Jobs of users with an LRAC or NRAC allocation will run at a higher priority than others while their resources last. Because of the group-based allocation, it is conceivable that your jobs won't run if your colleagues have already exhausted your group's limits.
Note that scheduling big jobs greatly affects the queuer and other users, so you have to talk to us first to run massively parallel jobs (> 2048 cores). We will help make sure that your jobs start and run efficiently.
If your job should run in fewer than 48 hours, specify that in your script -- your job will start sooner. (It's easier for the scheduler to fit in a short job than a long job). On the downside, the job will be killed automatically by the queue manager software at the end of the specified wallclock time, so if you guess wrong you might lose some work. So the standard procedure is to estimate how long your job will take and add 10% or so.
You interact with the queuing system through the queue/resource manager, Moab and Torque. To see all the jobs in the queue use
$ showq
To submit your own job, you must write a script which describes the job and how it is to be run (a sample script follows) and submit it to the queue, using the command
$ qsub [SCRIPT-FILE-NAME]
where you will replace [SCRIPT-FILE-NAME] with the file containing the submission script. This will return a job ID, for example 31415, which is used to identify the jobs. Information about a queued job can be found using
$ checkjob [JOB-ID]
and jobs can be canceled with the command
$ canceljob [JOB-ID]
Again, these commands have many options, which can be read about on their man pages.
Much more information on the queueing system is available on our queue page.
Batch Submission Script: MPI
A sample submission script is shown below for an mpi job with the #PBS directives at the top and the rest being what will be executed on the compute node.
<source lang="bash">
- !/bin/bash
- MOAB/Torque submission script for SciNet GPC
- PBS -l nodes=2:ppn=8,walltime=1:00:00
- PBS -N test
- DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR
- EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 16 ./a.out </source>
The lines that begin #PBS are commands that are parsed and interpreted by qsub at submission time, and control administrative things about your job. In this example, the script above requests two nodes, using 8 processors per node, for a wallclock time of one hour. (The resources required by the job are listed on the #PBS -l line.) Other options can be given in other #PBS lines, such as #PBS -N, which sets the name of the job.
The rest of the script is run as a bash script at run time. A bash shell on the first node of the two nodes that are requested executes these commands as a normal bash script, just as if you had run this as a shell script from the terminal. The only difference is that PBS sets certain environment variables that you can use in the script. $PBS_O_WORKDIR is set to be the directory that the command was 'submitted' from - eg, $SCRATCH/SOMEDIRECTORY. The script then uses the mpirun command to launch the job. Assumed here is that the user has a line like
$ module load openmpi intel
in their .bashrc (or one could put it in the job submission script itself anywhere before the mpirun.
- Note: The different versions of MPI require different commands to launch the run, and thus different scripts. The above script is specific for the openmpi module. For the intelmpi module on ethernet, the last line of the script should read
$ mpirun -np 16 -env I_MPI_FABRICS shm:tcp ./a.out
For full MPI details see MPI
Submitting Collections of Serial Jobs
You cannot run purely serial jobs on the GPC (or any of SciNet's systems), as this would mean only one core out of 8 is used. If you have serial jobs, you have to bunch them together. SciNet-approved methods for running collections of serial jobs can be found on the serial run wiki page.
Batch Submission Script: OpenMP
For running OpenMP jobs, the procedure is similar as for MPI jobs:
<source lang="bash">
- !/bin/bash
- MOAB/Torque submission script for SciNet GPC (OpenMP)
- PBS -l nodes=1:ppn=8,walltime=1:00:00
- PBS -N test
- DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8 ./a.out </source>
Note that in some circumstances it can be more efficient to run (say) two jobs each running on four threads than one job running on eight threads. In that case you can use the same `ampersand-and-wait' technique outlined for serial jobs (see serial run wiki page) for less-than-eight-core OpenMP jobs.
Hybrid MPI/OpenMP jobs
Using Intel MPI
Here is how to run hybrid codes using intelmpi::
http://software.intel.com/en-us/articles/hybrid-applications-intelmpi-openmp/
Make sure you compile with the -mt_mpi option to the compilers to use the thread safe libraries. Set the environment variable I_MPI_PIN_DOMAIN:
$ export I_MPI_PIN_DOMAIN=omp
This will set the process pinning domain size to be equal to OMP_NUM_THREADS (which you should set to the desired number of threads per mpi process). Therefore, each MPI process can create $OMP_NUM_THREADS number of children threads for running within the corresponding domain. If OMP_NUM_THREADS is not set, each node is treated as a separate domain (which will allow as many threads per MPI processes as there are cores).
In addition, when invoking mpirun, you should add the argument "-ppn X", where X is the number of MPI processes per node. For example:
$ mpirun -r ssh -ppn 2 -np 8 [executable]
would start 2 mpi processes of [executable] per node for a total of 8 processes, so mpirun will try to run mpi processes on 4 nodes (OMP_NUM_THREADS is then probably best set at 4). Your job script should still ask for these 4 nodes with the line <source lang="bash">
#PBS -l nodes=4:ppn=8,walltime=....
</source> (ppn=8 is not a mistake here; the ppn parameter has a different meaning for PBS and for mpirun)
The ppn parameter to mpirun is very important! Without it, eight mpi jobs would get bunched on the first node in this example, leaving 3 nodes unused.
NOTE: In order to pin OpenMP threads inside the domain, use the corresponding OpenMP feature by setting the KMP_AFFINITY environment variable, see Compiler User and Reference Guide.
The IntelMPI manual is referenced on the front page of our wiki:
http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf
For the above example of a total of 8 processes on 4 nodes, you could use the following script: <source lang="bash">
- !/bin/bash
- MOAB/Torque submission script for SciNet GPC (hybrid job)
- PBS -l nodes=4:ppn=8,walltime=1:00:00
- PBS -N test
- DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR
- SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4
- PIN THE MPI DOMAINS ACCORDING TO OMP
export I_MPI_PIN_DOMAIN=omp
- EXECUTION COMMAND; -np = nodes*ppn
mpirun -r ssh -ppn 2 -np 8 ./a.out </source>
Using Open MPI
For mixed MPI/OpenMP jobs using OpenMPI, which is the default for many users, the procedure is similar, but details differ.
- Request the number of nodes in the PBS script.
- Set OMP_NUM_THREADS to the number of threads per MPI process.
- In addition to the -np parameter for mpirun, add the argument --bynode, so that the mpi processes are not bunched up.
So for example, to start a total of 8 processes on 4 nodes, you could use the following script <source lang="bash">
- !/bin/bash
- MOAB/Torque submission script for SciNet GPC (hybrid job)
- PBS -l nodes=4:ppn=8,walltime=1:00:00
- PBS -N test
- DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR
- SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4
- EXECUTION COMMAND; -np = nodes*processes_per_nodes; --byhost forces a round robin of nodes.
mpirun -np 8 --bynode ./a.out </source>
- More questions about running your jobs? See the FAQs Running your jobs and Errors in running jobs sections.
Submitting an Interactive (Debug) Job
Your development work flow may require a lot of small test runs. You are allowed to do these on the development nodes, as long as it's very brief (a few minutes), and does not use all cores on the machine. For anything more you will have to use the compute nodes.
It is sometimes convenient to run a job interactively; this can be very handy for debugging purposes. In this case, you type a qsub command which submits an interactive job to the queue; when the scheduler selects this job to run, then it starts a shell running on the first node of the job, which connects to your terminal. You can then type any series of commands (for instance, the same commands listed as in the batch submission script above) to run a job interactively.
For example, to start the same sort of job as in the batch submission script above, but interactively, one would type
$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00
This is exactly the #PBS -l line in the batch script above (which requests all 8 processors on each of 2 nodes for one hour), but prepended with a -I for `interactive'. When this job begins, your terminal will now show you as being logged in to one of the compute nodes, and one can type in any shell command, run mpirun, etc. When you exit the shell, the job will end. Interactive jobs can be used with any of the GPC queues however, there is a short high turnover queue called debug which can be especially useful when the system is busy.
- More questions about running test jobs? See the FAQ.