User Serial

General considerations

So it should be said first that SciNet is a parallel computing resource, and our priority will always be parallel jobs. Having said that, if you can make efficient use of the resources using serial jobs and get good science done, that's good too, and we're happy to help you.

Nonetheless, you should never submit purely serial jobs to the queue (on either GPC or TCS). With our per-node scheduler, this would waste the computational power of 7 (or 31, on the TCS) cpus while the jobs is running. While we encourage you to try and parallelize your code, sometimes it is beneficial to run several serial codes at the same time. Note that because the TCS is specialized for parallel computing, you should only use the GPC for concurrent serial runs.

The GPC nodes each have 8 processing cores, and making efficient use of these nodes means using all eight cores. As a result, we'd like to have the users take up whole nodes by running 8 jobs or more at once.

When running multiple jobs on the same node, it is imperative to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes gpc01..04). So the jobs also have to be bunched in ways that will fit into 14GB. If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then its possible in principle to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at <support@scinet.utoronto.ca> and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the HyperThreading feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores has increased some users overall throughput by 10 to 30 percent.

Serial jobs of similar duration

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a bit like this <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for multiple serial jobs on
3. SciNet GPC
5. PBS -l nodes=1:ppn=8,walltime=1:00:00
6. PBS -N serialx8

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. EXECUTION COMMAND; ampersand off 8 jobs and wait

(cd jobdir1; ./dojob1) & (cd jobdir2; ./dojob2) & (cd jobdir3; ./dojob3) & (cd jobdir4; ./dojob4) & (cd jobdir5; ./dojob5) & (cd jobdir6; ./dojob6) & (cd jobdir7; ./dojob7) & (cd jobdir8; ./dojob8) & wait </source>

There are four important things to take note of here. First, the wait command at the end is crucial; without it the job will terminate immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they will take. If (say) dojob8 takes 2 hours and the rest only take 1, then for one hour 7 of the 8 cores on the GPC node are wasted; they are sitting idle but are unavailable for other users, and the utilization of this node over the whole run is only 56%. This is the sort of thing we'll notice, and users who don't make efficient use of the machine will have their ability to use scinet resources reduced. If you have many serial jobs of varying length, use the submission script to balance the computational load, as explained below.

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

GNU Parallel

You could also use GNU parallel in your script for the case above.

GNU parallel is a really nice tool to run multiple serial jobs in parallel. It offers essential the same functionality as the above on-node scripts, but with a syntax which is almost that of xargs.

GNU parallel is accessible on the GPC in the module gnu-parallel, which you can load in your .bashrc. <source lang="bash"> module load gnu-parallel </source>

It is easiest to demonstrate the usage of GNU parallel by examples. The script above can be replaced by <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for multiple serial jobs on SciNet GPC
4. PBS -l nodes=1:ppn=8,walltime=1:00:00
5. PBS -N serialx8

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. EXECUTION COMMAND

parallel -j 8 <<EOF

 cd jobdir2; ./dojob2
 cd jobdir3; ./dojob3
 cd jobdir4; ./dojob4
 cd jobdir5; ./dojob5
 cd jobdir6; ./dojob6
 cd jobdir7; ./dojob7
 cd jobdir8; ./dojob8

EOF </source>

The -j8 parameter sets the number of jobs to run at the same time.

For this particular case, using GNU Parallel or not is a matter of taste. The GNU-Parallel version is a bit more flexible though, since one could give, say, 32 command to the parallel command, which would be executed in bunched of eight automatically. And for the application below, the alternatives to GNU Parallel basically mean writing your own scheduler-within-a-scheduler, which is tricky and error prone.

Serial jobs of varying duration

If you have a lot (50+) of relatively short serial runs to do, of which the walltime varies, and if you know that eight jobs fit in memory without memory issues, then the following strategy in your submission script maximizes the cpu utilization. <source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for multiple, dynamically-run
3. serial jobs on SciNet GPC
5. PBS -l nodes=1:ppn=8,walltime=1:00:00
6. PBS -N serialdynamic

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*

find -name 'jobs*' | parallel -j 8 </source> Notes:

• GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is a simple way to do load balancing.
• You can run more or fewer than 8 processes per node by modifying parallel's -j8 argument.
• Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running off the ramdisk may be an option.
• When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
• More details on how to setup your script to use the ramdisk can be found on the Ramdisk wiki page.
• This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

Version for more than 8 cores at once (still serial)

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one gigE node and to use the following routine to distribute your processes amongst the cores.

<source lang="bash">

1. !/bin/bash
2. MOAB/Torque submission script for multiple, dynamically-run
3. serial jobs on SciNet GPC
5. PBS -l nodes=25:ppn=8,walltime=1:00:00
6. PBS -N serialdynamicMulti

1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

1. START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE

seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE -W$PWD ./myrun {}

1. Notes:
2. seq 800 : generates numbers 1 through 800 as input to parallel
3. -j8 : makes 8 commands run simultaneously on each node
4. --sshloginfile $PBS_NODEFILE : specifes the nodes to use
5. -W$PWD : start remote commands in current local directory
6. ./myrun {} : is the command to run, with {} replaced by the input

</source> Notes:

• Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
• GNU Parallel can be passed a file with the list of nodes to ssh to: This list is available in the fle $PBS_NODEFILE.
• Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
• GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to parallel, with {} replaces by the lines on input.
• The -W argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all!
• We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

More on GNU parallel

The documentation for GNU parallel can be found here and its man page here. The man page is also available on the GPC when the gnu-parallel module is loaded, with the command $ man parallel The main page contains options, such as how to make sure the output is not all scrambled, and examples.

Older scripts

Older scripts, which mimicked some of GNU parallel functionality, can be found on the Deprecated scripts page.

--Rzon 02:22, 2 April 2010 (UTC)