User Serial

From oldwiki.scinet.utoronto.ca
Jump to navigation Jump to search

General considerations

So it should be said first that SciNet is a parallel computing resource, and our priority will always be parallel jobs. Having said that, if you can make efficient use of the resources using serial jobs and get good science done, that's good too, and we're happy to help you.

Nonetheless, you should never submit purely serial jobs to the queue (on either GPC or TCS). With our per-node scheduler, this would waste the computational power of 7 (or 31, on the TCS) cpus while the jobs is running. While we encourage you to try and parallelize your code, sometimes it is beneficial to run several serial codes at the same time. Note that because the TCS is specialized for parallel computing, you should only use the GPC for concurrent serial runs.

The GPC nodes each have 8 processing cores, and making efficient use of these nodes means using all eight cores. As a result, we'd like to have the users take up whole nodes by running 8 jobs or more at once.

When running multiple jobs on the same node, it is imperative to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes gpc01..04). So the jobs also have to be bunched in ways that will fit into 14GB. If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then its possible in principle to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at <support@scinet.utoronto.ca> and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the HyperThreading feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores has increased some users overall throughput by 10 to 30 percent.

Serial jobs of similar duration

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a bit like this <source lang="bash">

  1. !/bin/bash
  2. MOAB/Torque submission script for multiple serial jobs on
  3. SciNet GPC
  5. PBS -l nodes=1:ppn=8,walltime=1:00:00
  6. PBS -N serialx8
  1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

  1. EXECUTION COMMAND; ampersand off 8 jobs and wait

(cd jobdir1; ./dojob1) & (cd jobdir2; ./dojob2) & (cd jobdir3; ./dojob3) & (cd jobdir4; ./dojob4) & (cd jobdir5; ./dojob5) & (cd jobdir6; ./dojob6) & (cd jobdir7; ./dojob7) & (cd jobdir8; ./dojob8) & wait </source>

There are four important things to take note of here. First, the wait command at the end is crucial; without it the job will terminate immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they will take. If (say) dojob8 takes 2 hours and the rest only take 1, then for one hour 7 of the 8 cores on the GPC node are wasted; they are sitting idle but are unavailable for other users, and the utilization of this node over the whole run is only 56%. This is the sort of thing we'll notice, and users who don't make efficient use of the machine will have their ability to use scinet resources reduced. If you have many serial jobs of varying length, use the submission script to balance the computational load, as explained below.

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

GNU Parallel

GNU parallel is a really nice tool to run multiple serial jobs in parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module gnu-parallel, which you can load in your .bashrc. <source lang="bash"> module load gnu-parallel </source>

It is easiest to demonstrate the usage of GNU parallel by examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script: <source lang="bash">

  1. !/bin/bash
  2. MOAB/Torque submission script for multiple serial jobs on SciNet GPC
  4. PBS -l nodes=1:ppn=8,walltime=24:00:00
  5. PBS -N gnu-parallel-example
  1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

module load gnu-parallel # or you can have this in your .bashrc

  1. EXECUTION COMMAND

parallel -j 8 <<EOF 

 cd jobdir1; ./dojob1; echo "job 1 finished"
 cd jobdir2; ./dojob2; echo "job 2 finished"
 cd jobdir3; ./dojob3; echo "job 3 finished"
 cd jobdir4; ./dojob4; echo "job 4 finished"
 cd jobdir5; ./dojob5; echo "job 5 finished"
 cd jobdir6; ./dojob6; echo "job 6 finished"
 cd jobdir7; ./dojob7; echo "job 7 finished"
 cd jobdir8; ./dojob8; echo "job 8 finished"
 cd jobdir9; ./dojob9; echo "job 9 finished"
 cd jobdir10; ./dojob10; echo "job 10 finished"
 cd jobdir11; ./dojob11; echo "job 11 finished"
 cd jobdir12; ./dojob12; echo "job 12 finished"
 cd jobdir13; ./dojob13; echo "job 13 finished"
 cd jobdir14; ./dojob14; echo "job 14 finished"
 cd jobdir15; ./dojob15; echo "job 15 finished"
 cd jobdir16; ./dojob16; echo "job 16 finished"

EOF </source>

The -j8 parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When a processor is done with a jobs, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

Serial jobs of varying duration

If you have a lot (50+) of relatively short serial runs to do, of which the walltime varies, and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization. <source lang="bash">

  1. !/bin/bash
  2. MOAB/Torque submission script for multiple, dynamically-run
  3. serial jobs on SciNet GPC
  5. PBS -l nodes=1:ppn=8,walltime=1:00:00
  6. PBS -N serialdynamic
  1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

module load gnu-parallel # or you can have this in your .bashrc

  1. COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*

find -name 'jobs*' | parallel -j 8 </source> Notes:

  • As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do load balancing.
  • You can in fact run more or less than 8 processes per node by modifying parallel's -j8 argument.
  • Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
  • When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
  • More details on how to setup your script to use the ramdisk can be found on the Ramdisk wiki page.
  • This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

Version for more than 8 cores at once (still serial)

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one gigE node and to use the following routine to distribute your processes amongst the cores.

<source lang="bash">

  1. !/bin/bash
  2. MOAB/Torque submission script for multiple, dynamically-run
  3. serial jobs on SciNet GPC
  5. PBS -l nodes=25:ppn=8,walltime=1:00:00
  6. PBS -N serialdynamicMulti
  1. DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from

cd $PBS_O_WORKDIR

  1. START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE

seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {} </source> Explanation:

  • seq 800 outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the parallel command.
  • The use of the "seq 800" is that each line that you give to parallel defines a new job. So here, there are 800 jobs.
  • Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument ./myrun {}. Here myrun is supposed to be the name of the application to run. The two curly brackets {} get replaced by the line from the input, that is, by one of the numbers.
  • So parallel will run the 800 commands:
    ./myrun 1
    ./myrun 2
    ...
    ./myrun 800
  • The parameter --sshloginfile $PBS_NODEFILE tells parallel to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
  • The parameter -j8 tells parallel to run 8 of these at a time on each of the hosts.
  • The --workdir $PWD sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail.

Notes:

  • Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
  • Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
  • GNU Parallel can be passed a file with the list of nodes to which to ssh, using --sshloginfile (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
  • Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
  • GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to parallel, with {} replaces by the lines on input.
  • The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all!
  • We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

More on GNU parallel

The documentation for GNU parallel can be found here and its man page here. The man page is also available on the GPC when the gnu-parallel module is loaded, with the command $ man parallel The main page contains options, such as how to make sure the output is not all scrambled, and examples.

Older scripts

Older scripts, which mimicked some of GNU parallel functionality, can be found on the Deprecated scripts page.

--Rzon 02:22, 2 April 2010 (UTC)

Retrieved from ‘https://oldwiki.scinet.utoronto.ca/index.php?title=User_Serial&oldid=4909’