oldwiki.scinet.utoronto.ca - User contributions [en-gb]

GPC Quickstart

2018-03-01T14:09:12Z

Nolta:

{{Infobox Computer
|image=[[Image:University_of_Tor_79284gm-a.jpg|center|300px|thumb]]
|name=General Purpose Cluster (GPC)
|installed=June 2009
|operatingsystem= Linux Centos 6.4
|loginnode= gpc01..gpc08 (from <tt>login.scinet</tt>)
|nnodes=3864 (30,912 cores)
|rampernode=16 Gb
|corespernode=8 (16 threads)
|interconnect=840 nodes 1:1 DDR, 3024 nodes 5:1 QDR
|vendorcompilers=icc (C) ifort (fortran) icpc (C++)
|queuetype=[[Moab | Moab/Torque]]
}}

'''The GPC will be decommissioned by early 2018. A new system for large parallel jobs, Niagara, will be replacing the [[GPC_Quickstart|GPC]] and [[TCS_Quickstart|TCS]]. To make room for the new system, the GPC has been reduced from 30,912 to 16,800 cores. Niagara is expected to be in production in April of 2018, at which time the GPC will be fully decommmissioned. '''

__TOC__

===Specifications===
The General Purpose Cluster is an extremely large cluster (ranked [http://www.top500.org/list/2009/06/100 16th] in the world at its inception, currently still at #391) and is where most computations are to be done at SciNet.
The GPC consists of 3,780 nodes with 16 GB of memory or more and 8 Intel cores each. The nodes use a shared parallel file system and have no local disks. The compute nodes are accessed through a queuing system that allows jobs with a minimum of 15 minutes and maximum wall time of 48 hours .

Technical details: The GPC is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the [http://www.hpcwire.com/features/HPC-Vendors-Jump-On-Nehalem-42360237.html first in the world] to make use of the new chips). The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,912 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking DDR InfiniBand while the rest of the nodes are connected with 5:1 blocked QDR InfiniBand.

Note: from an operational standpoint, the contributed systems [[Gravity]] and [[Sandy]], as well as the [[Visualization_Nodes|Viz]] system, are more or less part of the GPC.

===Login===

First login via [[Ssh | ssh]] with your SciNet account at <tt>login.scinet.utoronto.ca</tt>. This puts you on a 'login node', but not yet on the GPC. From the login node, you have to ssh to one of the 8 development nodes (<tt>gpc01</tt>, <tt>gpc02</tt>, ..., <tt>gpc08</tt>) to compile/test your code.

===Compile/Devel Nodes===

To get to one of the devel nodes from a scinet login node, you can ssh to <tt>gpc01</tt>..<tt>gpc08</tt>. You may also just use the <tt>gpc</tt> command to take directly to the dev node with lowest cpu load (reassessed every 5 minutes).

Except for these eight development nodes on the GPC, all other nodes are 'compute nodes' that can be used only through the scheduler. The devel nodes have the same hardware configuration as most of the compute nodes except with more memory -- 8 processing cores with 36 GB RAM and QDR Infiniband. You can compile and test your codes on these devel nodes. To interactively test on more than 8 processors, you can submit an [[GPC_Quickstart#Submitting_an_Interactive_.28Debug.29_Job | interactive job request]] to get time-limited command-line access to a compute node.

Your [[Storage_Quickstart | home directory]] is in <tt>$HOME</tt>. You have 50 GB there that is backed up. In addition, you have a larger, non-backed up space in <tt>$SCRATCH</tt> for temporary files. Note that your directories are encoded in the environment variables <tt>$HOME</tt> and <tt>$SCRATCH</tt>. Currently <tt>$HOME=/home/G/GROUP/USER</tt> and <tt>$SCRATCH=/scratch/G/GROUP/USER</tt> or <tt>/scratch2/G/GROUP/USER</tt> (where GROUP is your group's name, G is the first letter of that groups name, and USER is your user name). But the locations can change as the storage system evolves and grows with demand, so please use the environment variables!

The GPC devel and compute nodes do not have local disks. Instead, <tt>$HOME</tt> and <tt>$SCRATCH</tt> are shared parallel file systems (the file system is called GPFS), which means that your files are seen on all the nodes.

'''Your home directory cannot be written to by the compute nodes!''' Thus, to run jobs, you'll use the <tt>$SCRATCH</tt> directory (currently <tt>/scratch/g/group/USER</tt> but again, use the environment variable). Here, there is a large amount of disk space, but it is not backed up. Thus it makes sense to keep your codes in /home, compile there, and then run them in the /scratch directory.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including any of the compilers - , you will have to use the `module' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

Note that to use even the gcc compilers you will have to do
<pre>
$ module load gcc
</pre>
but in fact you probably should use the intel compilers installed on this system as they often produce faster executables (and occasionally, much faster.)

A list of the installed software and more information on using the module command is available in [[Software_and_Libraries | Software & Libraries]]. Or, when logged in to the gpc, you can
see all available modules on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load intel
</pre>
To unload a module
<pre>
$ module unload intel
</pre>
To unload all modules
<pre>
$ module purge
</pre>
To list all loaded modules
<pre>
$ module list
</pre>

These commands should go in your submission scripts to make sure you
are using the correct packages. It is possible to load them in your .bashrc files as well, but this is generally not recommended (see [[Important .bashrc guidelines]]), especially if you routinely have to flip back and forth between modules.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' affect other shell environments; in particular, a queued job that is running is unaffected by you interactively loading a module, and conversely you loading a module at the prompt and then submitting a job does not ensure that the module is loaded when the job runs. To ensure that a module is loaded when a job runs, be sure to put your <tt>module load</tt> command in your job submission script.

Again more information on modules, and how to resolve dependencies between modules can be found on the [[Software_and_Libraries | Software & Libraries]] page.

===Compilers===

The intel compilers are icc/icpc/ifort for C/C++/Fortran, and are available with the default module "intel". The intel compilers are recommended over the GNU compilers. Documentation about icpc is available at
http://software.intel.com/en-us/articles/intel-software-technical-documentation/. The Intel compilers accept many of the options that the GNU compilers accept, but tend to produce faster programs on our system. If, for some reason, you really need the GNU compilers, the latest version of the GNU compiler collection (currently 4.4.0) is available by loading the "gcc" module, with gcc/g++/gfortran for C/C++/Fortran. [[Co-array_Fortran_on_the_GPC|Coarray fortran]] is support by the intel compilers from version 12 upwards and by the GNU fortran compiler version 5.2.0. Note that the f77/g77 compilers are not supported, but the available fortran compilers are able to compile fortran 77.

To ensure that the intel compilers are in your <tt>PATH</tt> and their libraries are in your <tt>LD_LIBRARY_PATH</tt>, use the command

<pre>
$ module load intel/15.0.2
</pre>

Optimize your code for the GPC machine using of at least the following compiler flags:
<pre>
-O3 -xHost
</pre>
(or <tt>-O3 -march=native</tt> for the GNU compilers).

*If your program uses openmp, add <tt>-fopenmp</tt> for GNU compilers.
*If you get the warning <tt>feupdatreenv is not implemented</tt>, add -limf to the link line.
*If you need to link in the MKL libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code. '''Note that this give the link line for the command prompt. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''
*More questions about compiling? See the [[FAQ#Compiling_your_Code|FAQ]].

===Debuggers===

* '''ddt''' - Allinea's graphical parallel debugger, in the <tt>ddt</tt> module. Highly recommended!
* '''gdb''' - The GNU Debugger, available in the <tt>gdb</tt> module.
* '''idbc/idb''' - The intel debuggers, part of the <tt>intel</tt> module(s).
* '''ddd''' - A graphical debuggerThe GNU Debugger, available in the <tt>ddd</tt> module.

Note that to debug code, you have to give the <tt>-g</tt> flags to the compiler. The intel compiler needs the additional option <tt>-debug parallel</tt> to debug threaded/OpenMP code.

===[[ GPC_MPI_Versions | MPI]]===

SciNet currently provides multiple MPI libraries for the GPC; [http://www.open-mpi.org/ OpenMPI], and [http://software.intel.com/en-us/intel-mpi-library/ IntelMPI]. We currently recommend OpenMPI as the default, as it quite reliably demonstrates good performance on the infiniband network (and did so too on the ethernet network). For full details and options see the complete [[ GPC_MPI_Versions | '''MPI''']] section.

The MPI libraries are compiled with both the gnu compiler suite and the intel compiler suite. To use (for instance) the intel-compiled OpenMPI libraries, which we recommend as the default (and use for most of our examples here), use

<pre>
$ module load intel/15.0.2 openmpi/intel/1.6.4
</pre>

in your job submission scripts and on the command-line before compiling. Putting these in your <tt>.bashrc</tt> is no longer recommended (since Oct 10, 2013).

Other combinations behave similarly.

The MPI libraries define the wrappers mpicc/mpicxx/mpif90/mpif77 as wrappers around the appropriate compilers, which ensure the appropriate include and library directories are used in the compilation and linking steps.

We currently recommend the Intel + OpenMPI combination. However, if you require the GNU compilers as well as MPI, you will want to find the most recent openmpi module available with `gcc' in the version name. This will enable development and runtime with gcc/g++/gfortran and OpenMPI.


For mixed OpenMP/MPI code using Intel MPI, add the compilation flag -mt_mpi for full thread-safety (no such flag is necessary for OpenMPI).

===Submitting A Batch Job===

The SciNet machines are shared systems, and jobs that are to run on them are submitted to a queue; the
[[Moab | scheduler]] then orders the jobs in order to make the best use of the machine, and has them launched
when resources become availble. The intervention of the scheduler can mean that the jobs aren't
quite run in a first-in first-out order. The scheduler used on the GPC is called Moab (with Torque as the 'resource manager').

The scheduler will have the job run on one or more of the compute nodes of the GPC. There are a few important differences between the devel nodes and the compute nodes:

# As stated above already, on compute nodes, your home directory is read-only. You have to run your jobs from the $SCRATCH directory instead. See [https://support.scinet.utoronto.ca/wiki/index.php/Data_Management Data Management ] for more details on the file systems at SciNet.

# The available memory on compute nodes is approximately 14GB (16GB - 2GB for the operating system). The devel nodes have 36GB, but this is shared with all the users of the node.

# Some libraries, especially those for graphics, are not installed on the compute nodes. This leaves more memory available for your job, but if you have an application that requires such libraries (notably <tt>R</tt> and <tt>octave</tt>), you will need to "<tt>module load extras</tt>" in your job script to make them work.

The maximum [[wallclock time]] for a job in the queue is 48 hours; computations that will take longer than
this must be broken into 48-hour chunks and run as several jobs. Also a minimum job length of 15 minutes is enforced and shorter jobs
should be batched together. The usual way to do this is with [[checkpoints]],
writing out the complete state of the computation every so often in such a way that a job can be restarted from
this state information and continue on from where it left off. Generating [[checkpoints]] is a good idea anyway,
as in the unlikely event of a hardware failure during your run, it allows you to restart without having lost much work.

There are limits to how many jobs you can submit. If your group has a default account, up to 32 nodes at a time for 48 hours per job on the GPC cluster are allowed to be queued. This is a total limit, e.g., you could request 64 nodes for 24 hours. Jobs of users with an LRAC or NRAC allocation will run at a higher priority than others while their resources last. Because of the group-based allocation, it is conceivable that your jobs won't run if your colleagues have already exhausted your group's limits.

Note that scheduling big jobs greatly affects the queue and other users, so you have to talk to us first to run massively parallel jobs (> 2048 cores). We will help make sure that your jobs start and run efficiently.

If your job should run in fewer than 48 hours, specify that in your script -- your job
will start sooner. (It's easier for the [[Moab | scheduler]] to fit in a short job than a long job). On the downside, the
job will be killed automatically by the queue manager software at the end of the specified [[wallclock time]], so if you
guess wrong you might lose some work. So the standard procedure is to estimate how long your job will take and
add 10% or so.

You interact with the queuing system through the queue/resource manager, [[Moab | Moab]] and [[Moab | Torque]]. To see all the jobs in the queue use
<pre>
$ showq
</pre>

To submit your own job, you must write a script which describes the job and how it is to be run (a sample script [[GPC_Quickstart#Submission_Script | follows]]) and submit it to the queue, using the command
<pre>
$ qsub [SCRIPT-FILE-NAME]
</pre>
where you will replace <tt>[SCRIPT-FILE-NAME]</tt> with the file containing the submission script. This will return a job ID, for example 31415, which is used to identify the jobs. Information about a queued job can be found using
<pre>
$ qstat [JOB-ID]
</pre>
and jobs can be canceled with the command
<pre>
$ canceljob [JOB-ID]
</pre>

Again, these commands have many options, which can be read about on their man pages.

Much more information on the queueing system is available on our [[Moab | queue]] page.

====Batch Submission Script: MPI====

A sample submission script is shown below for an mpi job with the <tt> #PBS </tt> directives at the top and the rest being
what will be executed on the compute node.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC
#
#PBS -l nodes=2:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2 openmpi/intel/1.6.4

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 16 ./a.out
</source>

The lines that begin <tt>#PBS</tt> are commands that are parsed and interpreted by qsub at submission time, and control administrative things about your job. In this example, the script above requests two nodes, using 8 processors per node, for a [[wallclock time]] of one hour. (The resources required by the job are listed on the <tt>#PBS -l</tt> line.) Other options can be given in other <tt>#PBS</tt> lines, such as <tt>#PBS -N</tt>, which sets the name of the job.

The rest of the script is run as a bash script at run time. A bash shell on the first node of the two nodes that are requested executes these commands as a normal bash script, just as if you had run this as a shell script from the terminal. The only difference is that PBS sets certain environment variables that you can use in the script. <tt>$PBS_O_WORKDIR</tt> is set to be the directory that the command was 'submitted' from - eg, <tt>$SCRATCH/SOMEDIRECTORY</tt>. The script then uses the <tt>mpirun</tt> command to launch the job.



====Submitting Collections of Serial Jobs====

You cannot run purely serial jobs on the GPC (or any of SciNet's systems), as this would mean only one core out of 8 is used. If you have serial jobs, you have to bunch them together.
SciNet-approved methods for running collections of serial jobs can be found on the [[User_Serial|serial run wiki page]].

====Batch Submission Script: OpenMP====

For running OpenMP jobs, the procedure is similar as for MPI jobs:

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (OpenMP)
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=8
./a.out
</source>

Note that [[Introduction_To_Performance#Throughput | in some circumstances]] it can be more efficient to run (say) two jobs each running on four threads than one job running on eight threads. In that case you can use the same `ampersand-and-wait' technique outlined for serial jobs (see [[User_Serial|serial run wiki page]]) for less-than-eight-core OpenMP jobs.

====Hybrid MPI/OpenMP jobs====

'''Using Intel MPI'''

Here is how to run hybrid codes using intelmpi::

http://software.intel.com/en-us/articles/hybrid-applications-intelmpi-openmp/

Make sure you compile with the -mt_mpi option to the compilers to use the thread safe libraries.
Set the environment variable I_MPI_PIN_DOMAIN:
<pre>
$ export I_MPI_PIN_DOMAIN=omp
</pre>
This will set the process pinning domain size to be equal to OMP_NUM_THREADS (which you should set to the desired number of threads per mpi process). Therefore, each MPI process can create $OMP_NUM_THREADS number of children threads for running within the corresponding domain. If OMP_NUM_THREADS is not set, each node is treated as a separate domain (which will allow as many threads per MPI processes as there are cores).

In addition, when invoking mpirun, you should add the argument "-ppn X", where X is the number of MPI processes per node.
For example:
<pre>
$ mpirun -ppn 2 -np 8 [executable]
</pre>
would start 2 mpi processes of <tt>[executable]</tt> per node for a total of 8 processes, so mpirun will try to run mpi processes on 4 nodes
(OMP_NUM_THREADS is then probably best set at 4).
Your job script should still ask for these 4 nodes with the line
<source lang="bash">
#PBS -l nodes=4:ppn=8,walltime=....
</source>
(<tt>ppn=8</tt> is not a mistake here; the ppn parameter has a different meaning for PBS and for mpirun)

''The ppn parameter to ''mpirun'' is very important! Without it, eight mpi jobs would get bunched on the first node in this example, leaving 3 nodes unused.''

NOTE: In order to pin OpenMP threads inside the domain, use the corresponding OpenMP feature by setting the KMP_AFFINITY environment variable, see [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/optaps/common/optaps_openmp_thread_affinity.htm#KMP_AFFINITY_Environment_Variable|Intel's Compiler User and Reference Guide].

The IntelMPI manual is referenced on the front page of our wiki:

http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf

For the above example of a total of 8 processes on 4 nodes, you could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (hybrid job)
#
#PBS -l nodes=4:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2 intelmpi/4.1.2.040

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4

# PIN THE MPI DOMAINS ACCORDING TO OMP
export I_MPI_PIN_DOMAIN=omp

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -ppn 2 -np 8 ./a.out
</source>

'''Using Open MPI'''

For mixed MPI/OpenMP jobs using OpenMPI, which is the default for many users, the procedure is similar, but details differ.

* Request the number of nodes in the PBS script.
* Set OMP_NUM_THREADS to the number of threads per MPI process.
* In addition to the -np parameter for mpirun, add the argument <tt>--bynode</tt>, so that the mpi processes are not bunched up.

So for example, to start a total of 8 processes on 4 nodes, you could use the following script
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (hybrid job)
#
#PBS -l nodes=4:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2 openmpi/intel/1.6.4

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4

# EXECUTION COMMAND; -np = nodes*processes_per_nodes; --byhost forces a round robin of nodes.
mpirun -np 8 --bynode ./a.out
</source>

* More questions about running your jobs? See the [[FAQ#Running_your_jobs|FAQs Running your jobs]] and [[FAQ#Errors_in_running_jobs|Errors in running jobs]] sections.

===Automated Email Notifications from Jobs===

By default, you get an email if your job fails ('''a'''borts) when run through the scheduler. You can setup email notifications as well for when the jobs '''b'''egins or '''e'''nds. For this, you use the <tt>-m</tt> submission option, followed by any combination of '''a''' (to get emails when the job is aborted, e.g. by canceljob or qdel), '''b''' (to get emails when the job starts), and '''e''' (to get emails when the job ends), or '''n''' (for no emails except errors such as going over the walltime limit).

For instance, to get email in all three cases, you could have a job script like this:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC
#
#PBS -l nodes=2:ppn=8,walltime=1:00:00
#PBS -N test
#PBS -m abe

# load modules (must match modules used for compilation)
module load intel openmpi

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 16 ./a.out
</source>

Notes:

* Getting email for every job is not necessarily a good idea, as you may get a lot of email. This is why the default is to send an email only when the job fails

* The email address that the emails are sent to is the one that you gave when you created your account.


* Emails notifying that a job exceeded its walltime cannot be suppressed. The only way not to get them is to have your jobs stop before the walltime. If your job uses [[Checkpoints]] properly so that it can restart, it may be safe to have that job script self-terminate just before the walltime is over, by adding, at the top of the script, a line like this:

<source lang="bash">
sleep $((PBS_WALLTIME-60)) && (echo "out of time, self-terminating"; kill $$) &
</source>
which terminates the jobs script one minute before the wall time is over. You may want to combine this technique with [[Using Signals]].

===Submitting an Interactive (Debug) Job===

Your development work flow may require a lot of small test runs. You are allowed to do these on the development nodes, as long as it's very brief (a few minutes), and does not use all cores on the machine. For anything more you will have to use the compute nodes. (In a hurry? Check out the debugjob command below!)

It is sometimes convenient to run a job interactively; this can be very handy for debugging purposes. In this case, you type a <tt>qsub</tt> command which submits an interactive job to the queue; when the scheduler selects this job to run, then it starts a shell running on the first node of the job, which connects to your terminal. You can then type any series of commands (for instance, the same commands listed as in the batch submission script above) to run a job interactively.

For example, to start the same sort of job as in the batch submission script above, but interactively, one would type
<pre>
$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00
</pre>
If your interactive job requires graphics, add "-X" to this command.

Note that this is exactly the <tt>#PBS -l</tt> line in the batch script above (which requests all 8 processors on each of 2 nodes for one hour), but prepended with a <tt>-I</tt> for `interactive'. When this job begins, your terminal will now show you as being logged in to one of the compute nodes, and one can type in any shell command, run <tt>mpirun</tt>, etc. When you exit the shell, the job will end.

Interactive jobs can be used with any of the [[ Moab#GPC | GPC queues ]] however, there is a short
high turnover queue called [[ Moab#debug | debug ]] which can be especially useful when the system is busy:
<pre>
$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00 -q debug
</pre>

Because the combination of running interactively and use the high turnover queue is very common, SciNet has the specialized command
<pre>
$ debugjob [NUMBEROFNODES]
</pre>
where <tt>NUMBEROFNODES</tt> is an optional argument that defaults to one. This command requests NUMBEROFNODES compute nodes for a limited time between 30 minutes and 2 hours (less time the more nodes you take), and gives you a shell on the (first) compute node.

*More questions about running test jobs? See the [[FAQ#Testing_your_Code|FAQ]].



===QDR vs. DDR Infiniband===

The GPC Infiniband network GPC has two sections, one connecting 3,024 nodes with 5:1 blocking/oversubscribed QDR and the second connecting 840 nodes 1:1 non-blocking DDR Infiniband. By default a user's job will go to whichever network section best accommodates it, typically smaller jobs to the QDR and larger jobs to the DDR. However a user can override this by simply adding the flags "ddr" or "qdr" to the job resource request.

For example, to request two nodes anywhere on the GPC (QDR or DDR), use
<pre>
#PBS -l nodes=2:ppn=8,walltime=1:00:00
</pre>
in your job submission script.

For two nodes using DDR, use
<pre>
#PBS -l nodes=2:ddr:ppn=8,walltime=1:00:00
</pre>

To get two nodes using QDR, instead, you would use
<pre>
#PBS -l nodes=2:qdr:ppn=8,walltime=1:00:00
</pre>

The queueing system also tries its best to keep jobs within the same switch of the QDR thus avoiding the 5:1 blocking. A user can can explicitly request this behaviour if their jobs is less than 30 nodes using

<pre>
#PBS -W x=nodesetisoptional:false
</pre>

===HyperThreading===

Each GPC compute node has 8 Nehalem cores (2 sockets each with a four-core Intel Xeon E5540 @ 2.53GHz). Thus, to make full use of the computing power of a GPC node, you must be running least 8 "tasks" -- MPI processes, or OpenMP threads.

Under most circumstances, running exactly 8 tasks is the most efficient way to use these nodes. However, sometimes software design (eg, having one thread for communication and one for computation) can usefully `oversubscribe' the number of physical cores, and running (say) twice as many tasks as cores can be a useful strategy. If your code is highly memory-bandwidth bound, having one task ready to run while another waits for memory access can make more effective use of the processor.

The Nehalem processors have hardware support for such two-way overloading of processors, through "HyperThreading"; there are an extra set of registers on each core to facilitate rapid switching between two tasks, making it look to the operating system that there are in fact 16 cores per node. Depending on the nature of your code, making use of these virtual extra cores may speed up or slow down your computation; you should run small test cases before running production jobs in this manner. In most cases, the speed difference will be under 10%. Some of our users have obtained an 8% speedup by running gromacs with 16 tasks instead of 8 on a single node (mpirun -np 16 ./gromacs/mdrun -npme 4 is 108% the speed of mpirun -np 8 ./gromacs/mdrun with -npme 2 or -1).

====HyperThreading with OpenMP====

To use hyperthreading with an OpenMP job, one just runs twice as many threads as one would have previously; eg, if you were running 8 threads before (<tt>export OMP_NUM_THREADS=8</tt>) you would run with 16 (<tt>export OMP_NUM_THREADS=16</tt>). Everything else remains the same, including the job submission script; one still uses <tt>ppn=8</tt> in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

====HyperThreading with MPI====

To use hyperthreading with an MPI job, one just runs twice as many MPI processes as one would have previously; eg, if you were running on three nodes using 8 MPI tasks per node and used <tt>mpirun ... -np 24</tt>, you could run instead with <tt>-np 48</tt>. Everything else remains the same, including the job submission script; one still uses <tt>ppn=8</tt> in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

Note that if you are using OpenMPI (as is the default), there is another consideration; OpenMPI assumes that there is no oversubscription and each task very aggressively makes full use of a core when it is waiting for a message (eg, the waits are "busywaits"). If you find a significant slowdown when running multiple MPI tasks per core with OpenMPI, you may want to try adding the additional option to mpirun: <tt>--mca mpi_yield_when_idle 1</tt>. This will increase the latency of individual messages, but free up the core to do additional work while waiting.

With IntelMPI, the problem should be less pronounced, but you can still improve things by using <tt>mpirun -genv I_MPI_SPIN_COUNT 1 ...</tt>

'''Examples of hyperthreading with MPI'''

Hyperthreading using gromacs: https://support.scinet.utoronto.ca/wiki/index.php/Gromacs#Hyperthreading_with_Gromacs

====HyperThreading with Hybrid MPI/OpenMP codes====

With a hybrid code, one has extra flexibility in how to assign the "extra" cores -- you could run extra MPI tasks or extra OpenMPI threads. As with all hybrid codes, the combination which results in the best performance depends very strongly on the nature of your code, and you should experiment with different combinations. In addition, with hybrid codes processor and memory affinity issues become very important; if you're unsure as to how to tune your application for best performance, please make an appointment with the SciNet technical analysts for more help.

===Memory Configuration===

{| class="wikitable"
!{{Hl2}}|Number of Nodes
!{{Hl2}}|Memory
!{{Hl2}}|Notes
|-
|3655
|16 GB
|14 GB is available for your application.
|-
|205
|32 GB
|use the flag m32g to request (see below)
|-
|72
|64 GB
|part of the contributed system [[Sandy]], but unused cycles are available.
|-
|4
|128 GB or 256 GB
|2 x 128 GB and 2x256 GB as part of [[Sandy]],
|-
|2
|128 GB
|2 x older harpertown nodes with 16 cores and 128 GB each.
|-
|2
|128 GB
|2 x new haswell nodes with 20 cores and 128 GB each.
|}

'''16G'''

There are 3655 nodes which have 16G of memory, and is the primary configuration in the GPC. These nodes will be used by default.
On these nodes, about 2 GB is taken by the operating system. So for mpi runs with 8 processes per node, this leaves about 1.75GB max per mpi process. Do not try to use more than the available memory: the node will crash and your job will either die or hang until the requested walltime has elapsed.

If you need more memory per process or per thread, you can either try to use the limit number of larger memory nodes listed below, or you can run with less mpi processes, or use a different decomposition, such that the job fits on a node.



'''32G'''

There are 205 nodes which have 32G of memory. To request these nodes use:

<pre>
$ qsub -l nodes=1:m32g:ppn=8,walltime=1:00:00
</pre>

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total of 32GB.

'''64G/128G/256G'''

There are 72 16-core Intel Sandybridge nodes which have 64G, 2 with 128G, and 2 with 256G of memory available as part of the contributed [[Sandy]] cluster. These nodes are requested through the <tt>sandy</tt> queue.

<pre>
$ qsub -l nodes=1:m64g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m256g:ppn=16,walltime=1:00:00 -q sandy
</pre>

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total.

'''128G'''

There are four other stand-alone large memory (128GB) nodes which are primarily to be used for data analysis of runs. Both are Intel machines running the same linux OS as the compute nodes however, they are of different Intel generations than the regular GPC compute nodes. Two nodes are 16 core older Harpertown nodes, so codes may have to be compiled separately for these machine, and some modules that work on the other GPC nodes, such as octave, will not work on these nodes. The two other are 20 core newer Haswell nodes that all of the GPC modules should work on.

These nodes can be accessed using a specific <tt>largemem</tt> queue

<pre>
$ qsub -l nodes=1,walltime=1:00:00 -q largemem -I
</pre>

To specifically request the older/newer nodes, use the number of processers (16/20)

<pre>
$ qsub -l nodes=1:ppn=16,walltime=1:00:00 -q largemem -I
$ qsub -l nodes=1:ppn=20,walltime=1:00:00 -q largemem -I
</pre>

Use the latter reqeust to run any code compiled specifically for the GPC nodes (such as the python modules).

'''Note:''' To estimate your time of access to these nodes, use

<pre>
$ showq -w class=largemem
</pre>

====How to run fewer MPI processes per node for memory bound applications====

If your compute job requires more memory than is available, it will crash. In some cases, this will even crash the node itself.
One way to exceed the memory is if your MPI application is too large for 8 processes to fit in memory. Oftentimes, you can solve this problem by subdividing the computation over more processes (reducing the memory per process) and ask for more nodes. Sometimes, this does not work, for instance if each MPI process requires a fixed minimum chunk of memory that is too large. In that case, you will want to run with less mpi processes than there are cpus in a nodes (i.e. less than 8 for the GPC). Running in such a mode requires a change in the mpirun command. The mpirun command now should not only specify the total number of mpi processes, but should also specify how these processes will be distributed over the nodes. The default distribution of processes is such that the first nodes get loaded with 8 processes, than the second nodes gets filled up, etc, and this will clearly lead to the same memory issue that you had before. Instead, you would want your processes to be distributed evenly over the nodes. The syntax for specifying that depends on whether you are using openmpi or intelmpi. For openmpi running on 4 nodes with 3 processes per node, for example, you would invoke mpirun as follows
<pre>
mpirun -np 12 --bynode <application>
</pre>
For intelmpi, the same setup would be invoked using:
<pre>
mpirun -np 12 -ppn 3 <application>
</pre>

====[https://wiki.scinet.utoronto.ca/wiki/index.php/Moab#Checking_the_memory_usage_from_jobs Checking memory usage from jobs]====

===Ram Disk===

On the GPC nodes, there is a `ram disk' available - up to half of the memory on the node may be used as a temporary file system. This is particularly useful for use in the early stages of migrating destop-computing codes to a High Performance Computing platform such as the GPC. It is much faster than real disk and does not require network traffic; however, each node sees its own ramdisk and cannot see files on that of other nodes. This is a very easy way to cache writes (by writing them to fast ram disk instead of slow `real' disk); and then one would periodically copy the files to files on /scratch or /project so that they are available after the job has completed.

To use the ramdisk, create and read to or write from files in /dev/shm/... just as one would to (eg) $SCRATCH. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

NOTE: it is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].

=== Managing jobs on the Queuing system ===

Information on checking available resources, starting, viewing, managing and canceling jobs on [[Moab | Moab/Torque]]. Also check out the [[FAQ#Monitoring_jobs_in_the_queue|FAQ]].

Oldwiki.scinet.utoronto.ca:System Alerts

2017-11-12T16:12:17Z

Nolta:

Oldwiki.scinet.utoronto.ca:System Alerts

2017-11-12T16:04:43Z

Nolta: /* System Status */

Ansys

2017-06-12T17:46:57Z

Nolta:

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

=Getting a license=

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,
and give them your SciNet username.

=Ansys 18.1=

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

==Running Fluent==

Example submission script for a job running on 4 nodes, with max walltime of 24 hours:

<source lang="bash">
#!/bin/bash
#PBS -l nodes=4:ppn=8,walltime=24:00:00

module load ANSYS/18.1 extras

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

INPUTFILE=input.jou
fluent 2ddp -t "$PBS_NP" -cnf="$PBS_NODEFILE" -mpi=intel -pib -pcheck -g -i "$INPUTFILE"
</source>

Ansys

2017-05-30T19:01:41Z

Nolta: /* Running Fluent */

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

=Getting a license=

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,
and give them your SciNet username.

=Ansys 18.1=

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

==Running Fluent==

Example submission script for a job running on 4 nodes, with max walltime of 24 hours:

<source lang="bash">
#!/bin/bash
#PBS -l nodes=4:ppn=8,walltime=24:00:00

module load ANSYS/18.1

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

INPUTFILE=input.jou
fluent 2ddp -t "$PBS_NP" -cnf="$PBS_NODEFILE" -mpi=intel -pib -pcheck -g -i "$INPUTFILE"
</source>

Ansys

2017-05-30T18:59:29Z

Nolta: /* Running Fluent */

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

=Getting a license=

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,
and give them your SciNet username.

=Ansys 18.1=

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

==Running Fluent==

Example running on 4 nodes, with max walltime of 24 hours:

<source lang="bash">
#!/bin/bash
#PBS -l nodes=4:ppn=8,walltime=24:00:00

module load ANSYS/18.1

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

INPUTFILE=input.jou
fluent 2ddp -t "$PBS_NP" -cnf="$PBS_NODEFILE" -mpi=intel -pib -pcheck -g -i "$INPUTFILE"
</source>

Ansys

2017-05-30T18:59:01Z

Nolta: /* Running Fluent */

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

=Getting a license=

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,
and give them your SciNet username.

=Ansys 18.1=

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

==Running Fluent==

Example running on 4 nodes, with max walltime of 24 hours:

<source lang="bash">
#!/bin/bash
#PBS -l nodes=4:ppn=8,walltime=24:00:00

module load ANSYS/18.1

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

INPUTFILE=input.jou
fluent 2ddp -t "$PBS_NP" -cnf="$PBS_NODEFILE" -mpi=intel -pib -pcheck -g -i $INPUTFILE
</source>

Ansys

2017-05-30T18:56:01Z

Nolta: /* Running Fluent */

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

=Getting a license=

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,
and give them your SciNet username.

=Ansys 18.1=

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

==Running Fluent==

Example running on 4 nodes, with max walltime of 24 hours:

<source lang="bash">
#!/bin/bash
#PBS -l nodes=4:ppn=8,walltime=24:00:00

module load ANSYS/18.1

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

fluent 2ddp -t "$PBS_NP" -cnf="$PBS_NODEFILE" -mpi=intel -pib -pcheck -g -i input.jou
</source>

Ansys

2017-05-30T18:01:33Z

Nolta:

Ansys

2017-05-30T17:58:24Z

Nolta: /* Getting a license */

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

==Getting a license==

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,
and give them your SciNet username.

==Ansys 18.1==

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

Ansys

2017-05-30T17:51:57Z

Nolta: Created page with "The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet. ==Getting a license== Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]...."

The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.

==Getting a license==

Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx

==Ansys 18.1==

module load ANSYS/18.1

Programs available:

* fluent
* ansysedt
* mapdl
* ...

User Codes

2017-05-30T17:20:23Z

Nolta:

__FORCETOC__

==Astrophysics==

===Athena (explicit, uniform grid MHD code)===

[[Image:StrongScalingAthenaGPC.png|thumb|right|320px|Athena scaling on GPC with OpenMPI and MVAPICH2 on GigE, and OpenMPI on InfiniBand]]

[http://www.astro.princeton.edu/~jstone/athena.html Athena] is a straightforward C code which doesn't use a lot of libraries so it is pretty straightforward to build and compile on new machines.

It encapsulates its compiler flags, etc in an <tt>Makeoptions.in</tt> file which is then processed by <tt>configure</tt>. I've used the following additions to <tt>Makeoptions.in</tt> on TCS and GPC:

<source lang="make">
ifeq ($(MACHINE),scinettcs)
CC = mpcc_r
LDR = mpcc_r
OPT = -O5 -q64 -qarch=pwr6 -qtune=pwr6 -qcache=auto -qlargepage -qstrict
MPIINC =
MPILIB =
CFLAGS = $(OPT)
LIB = -ldl -lm
else
ifeq ($(MACHINE),scinetgpc)
CC = mpicc
LDR = mpicc
OPT = -O3
MPIINC =
MPILIB =
CFLAGS = $(OPT)
LIB = -lm
else
...
endif
endif
</source>
It performs quite well on the GPC, scaling extremely well even on a strong scaling test out to about 256 cores (32 nodes) on Gigabit ethernet, and performing beautifully on InfiniBand out to 512 cores (64 nodes).

-- [[User:Ljdursi|ljdursi]] 19:20, 13 August 2009 (UTC)

===FLASH3 (Adaptive Mesh reactive hydrodynamics; explict hydro/MHD)===

[[Image:weak-scaling-example.png|thumb|right|320px|Weak scaling test of the 2d sod problem on both the GPC and TCS. The results are actually somewhat faster on the GPC; in both cases (weak) scaling is very good out at least to 256 cores]]

[http://flash.uchicago.edu FLASH] encapsulates its machine-dependant information in the <tt>FLASH3/sites</tt> directory. For the GPC, you'll have to
<pre>
module load intel
module load openmpi
module load hdf5/184-p1-v16-openmpi
</pre>

and with that, the following file (<tt>sites/scinetgpc/Makefile.h</tt>) works for me:
<source lang="make">
## Must do module load hdf5/183-v16-openmpi
HDF5_PATH = ${SCINET_HDF5_BASE}
ZLIB_PATH = /usr/local

#----------------------------------------------------------------------------
# Compiler and linker commands
#
# We use the f90 compiler as the linker, so some C libraries may explicitly
# need to be added into the link line.
#----------------------------------------------------------------------------

## modules will put the right mpi in our path
FCOMP = mpif77
CCOMP = mpicc
CPPCOMP = mpiCC
LINK = mpif77

#----------------------------------------------------------------------------
# Compilation flags
#
# Three sets of compilation/linking flags are defined: one for optimized
# code, one for testing, and one for debugging. The default is to use the
# _OPT version. Specifying -debug to setup will pick the _DEBUG version,
# these should enable bounds checking. Specifying -test is used for
# flash_test, and is set for quick code generation, and (sometimes)
# profiling. The Makefile generated by setup will assign the generic token
# (ex. FFLAGS) to the proper set of flags (ex. FFLAGS_OPT).
#----------------------------------------------------------------------------

FFLAGS_OPT = -c -r8 -i4 -O3 -xSSE4.2
FFLAGS_DEBUG = -c -g -r8 -i4 -O0
FFLAGS_TEST = -c -r8 -i4

LIB_HDF5 = -L${HDF5_PATH}/lib -lhdf5 -L${SCINET_ZLIB_LIB} -lz -lgpfs

# if we are using HDF5, we need to specify the path to the include files
CFLAGS_HDF5 = -I${HDF5_PATH}/include

CFLAGS_OPT = -c -O3 -xSSE4.2
CFLAGS_TEST = -c -O2
CFLAGS_DEBUG = -c -g

MDEFS =

.SUFFIXES: .o .c .f .F .h .fh .F90 .f90

#----------------------------------------------------------------------------
# Linker flags
#
# There is a seperate version of the linker flags for each of the _OPT,
# _DEBUG, and _TEST cases.
#----------------------------------------------------------------------------

LFLAGS_OPT = -o
LFLAGS_TEST = -o
LFLAGS_DEBUG = -g -o

MACHOBJ =

MV = mv -f
AR = ar -r
RM = rm -f
CD = cd
RL = ranlib
ECHO = echo
</source>

-- [[User:Ljdursi|ljdursi]] 22:11, 13 August 2009 (UTC)

==Aeronautics==

==Chemistry==

===CPMD===

Please refer to the [[Cpmd | CPMD]] page.

===NWChem===

Please refer to the [[Nwchem | NWChem]] page.

===GAMESS (US)===

Please refer to the [[gamess|GAMESS (US)]] page.

User supplied content below.

====Tips from the Fekl Lab====

Through trial and error, we have found a few useful things that we would like to share:

1. Two very useful, open-source programs for visualization of output files from GAMESS(US) and for generation of input files are [http://www.scl.ameslab.gov/MacMolPlt/ MacMolPlt]and [http://avogadro.openmolecules.net/wiki/Main_Page Avogadro]. The are available for UNIX/LINUX, Windows and Mac based machines, HOWEVER: any input files that we have generated with these programs on a Windows-based machine do not run on Mac based machines. We don't know why.

2. [http://winscp.net/eng/index.php WinSCP] is a very useful tool that has a graphical user interface for moving files from a local machine to SCINET and vice versa. It also has text editing capabilities.

3. The [https://bse.pnl.gov/bse/portal ESML Basis Set Exchange] is an excellent source for custom basis set or effective core potential parameters. Make sure that you specify "Gamess-US" in the format drop-down box.

4. The commercial program [http://www.chemcraftprog.com/ ChemCraft] is a highly useful visualization program that has the ability to edit molecules in a very similar fashion to GaussView. It can also be customized to build GAMESS(US) input files.

====Anatomy of a GAMESS(US) Input File with Basis Set Info in an External File====

$CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE DFTTYP=M06-L MAXIT=199 MULT=1 NOSYM=1
ECP=READ $END
$SYSTEM TIMLIM=525600 MWORDS=1750 PARALL=.TRUE. $END
$BASIS GBASIS=CUSTOMNI EXTFIL=.t. $END
$SCF DIRSCF=.TRUE. FDIFF=.f. $END
$STATPT OPTTOL=0.0001 NSTEP=500 HSSEND=.t. $END
$DATA
Mo_BDT3
C1
MOLYBDENUM 42.0 5.7556500000 4.4039600000 16.5808400000
SULFUR 16.0 7.4169700000 3.1956300000 15.2089300000
SULFUR 16.0 4.0966800000 3.2258300000 15.1761100000
SULFUR 16.0 3.9677300000 4.4940500000 18.3266100000
SULFUR 16.0 7.1776900000 3.5815000000 18.4485200000
SULFUR 16.0 4.3776600000 6.2447400000 15.6786900000
SULFUR 16.0 7.5478700000 6.0679800000 16.2223700000
CARBON 6.0 6.4716900000 2.1004800000 14.1902300000
CARBON 6.0 5.0690300000 2.1781400000 14.1080700000
CARBON 6.0 4.8421800000 4.2701300000 19.8855500000
CARBON 6.0 6.1969000000 3.9249600000 19.9397400000
CARBON 6.0 6.8280600000 3.7834200000 21.1913200000
CARBON 6.0 5.7697600000 7.6933500000 17.4241800000
CARBON 6.0 7.2043100000 7.9413600000 17.8281100000
CARBON 6.0 5.5051400000 7.0409700000 14.5903800000
CARBON 6.0 6.8905200000 6.9194700000 14.7626200000
CARBON 6.0 7.7396400000 7.5379800000 13.8285700000
HYDROGEN 1.0 8.8190700000 7.4520600000 13.9252200000
CARBON 6.0 7.2169400000 8.2960300000 12.7704100000
HYDROGEN 1.0 7.8667000000 8.7825100000 12.0575600000
CARBON 6.0 5.8260300000 8.4502300000 12.6467800000
HYDROGEN 1.0 5.4143000000 9.0544300000 11.8493100000
CARBON 6.0 4.9881500000 7.8192300000 13.5528400000
HYDROGEN 1.0 3.9090500000 7.9420000000 13.4583700000
CARBON 6.0 7.1538500000 1.1569600000 13.4143900000
CARBON 6.0 4.4018100000 1.3603900000 13.1919900000
CARBON 6.0 6.4791600000 0.3185500000 12.5353300000
CARBON 6.0 5.0837400000 0.4369500000 12.4084900000
HYDROGEN 1.0 7.0116000000 -0.4099400000 11.9434600000
HYDROGEN 1.0 8.2399000000 1.0702400000 13.4937600000
HYDROGEN 1.0 3.3185600000 1.4368700000 13.0953100000
HYDROGEN 1.0 4.5549800000 -0.1997300000 11.7165200000
CARBON 6.0 6.1105700000 3.9639000000 22.3866100000
CARBON 6.0 4.1216300000 4.4424400000 21.1020100000
HYDROGEN 1.0 7.8732900000 3.5217100000 21.2520500000
CARBON 6.0 4.7606000000 4.2868500000 22.3363800000
HYDROGEN 1.0 6.6064200000 3.8406000000 23.3428500000
HYDROGEN 1.0 4.2065000000 4.4170700000 23.2667100000
HYDROGEN 1.0 3.0674000000 4.6893500000 21.0889000000
HYDROGEN 1.0 7.4249200000 7.7545300000 18.8583200000
HYDROGEN 1.0 7.6651700000 8.9049700000 17.7652100000
HYDROGEN 1.0 5.3324000000 8.6487800000 17.2222700000
HYDROGEN 1.0 5.5015000000 7.1039000000 18.2759400000
$END
$ECP
MO-ECP GEN 28 3
5 ----- f potential -----
-0.0469492 0 537.9667807
-20.2080084 1 147.8982938
-106.2116302 2 45.7358898
-41.8107368 2 13.2911467
-4.2054103 2 4.7059961
3 ----- s-f potential -----
2.8063717 0 110.2991760
44.5162012 1 23.2014645
82.7785227 2 5.3530131
4 ----- p-f potential -----
4.9420876 0 63.2901397
25.8604976 1 23.3315302
132.4708742 2 24.6759423
57.3149794 2 4.6493040
5 ----- d-f potential -----
3.0054591 0 104.4839977
26.3637851 1 66.2307245
183.3849199 2 39.1283176
98.4453068 2 13.1164437
22.4901377 2 3.6280263
S NONE
S NONE
S NONE
S NONE
S NONE
S NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
H NONE
C NONE
H NONE
C NONE
H NONE
C NONE
H NONE
C NONE
C NONE
C NONE
C NONE
H NONE
H NONE
H NONE
H NONE
C NONE
C NONE
H NONE
C NONE
H NONE
H NONE
H NONE
H NONE
H NONE
H NONE
H NONE
$END

=====The Input Deck=====

Below is the input deck. It is where you tell GAMESS(US) what job type to execute and where all you individual parameters are entered for your specific job type. The example input deck below is for a geometry optimization and frequency calculation. This input deck is equivalent to the Gaussian job with "opt" and "freq" in the route section.

$CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE DFTTYP=M06-L MAXIT=199 MULT=1 NOSYM=1
ECP=READ $END
$SYSTEM TIMLIM=2850 MWORDS=1750 MEMDDI=20 PARALL=.TRUE. $END
$BASIS GBASIS=CUSTOMNI EXTFIL=.t. $END
$SCF DIRSCF=.TRUE. FDIFF=.f. $END
$STATPT OPTTOL=0.0001 NSTEP=500 HSSEND=.t. $END
$DATA

An important thing to note is the spacing. In the input deck, there must be 1 space at the beginning of each line of the input deck. If not, the job will fail. Most builders will insert this space anyway, but it helps to double check.

The end of the input deck is marked by the "$DATA" line.

=====Job Title Line=====

The next line of the file is the job title. It can be anthing you wish, however, we have found that to be on the safe side, we avoide using symbols or spaces

Mo_BDT3

=====Symmetry Point Group=====

The next line of the file is the symmetry point group of your molecule. Note that there is no leading space before the point group.

C1

=====Coordinates=====

The next block of text is set aside for the coordinates of the molecule. This can be in internal (or z-matrix) format or cartesian coordinates. Note that there is no leading space before the coordinates. One may use the chemical symbol or the full name of each atom in the molecule. Note that the end of the coordinates is signified by an "$END", which MUST have one space preceding it. The coordinates below do NOT have any basis set information inserted. It is possible to insert basis set information directly into the input file. This is accomplished by obtaining the desired basis set parameters from the EMSL and then inserting them below each relevant atom. An example input file with inserted basis set information will be shown later.

MOLYBDENUM 42.0 5.7556500000 4.4039600000 16.5808400000
SULFUR 16.0 7.4169700000 3.1956300000 15.2089300000
SULFUR 16.0 4.0966800000 3.2258300000 15.1761100000
SULFUR 16.0 3.9677300000 4.4940500000 18.3266100000
SULFUR 16.0 7.1776900000 3.5815000000 18.4485200000
SULFUR 16.0 4.3776600000 6.2447400000 15.6786900000
SULFUR 16.0 7.5478700000 6.0679800000 16.2223700000
CARBON 6.0 6.4716900000 2.1004800000 14.1902300000
CARBON 6.0 5.0690300000 2.1781400000 14.1080700000
CARBON 6.0 4.8421800000 4.2701300000 19.8855500000
CARBON 6.0 6.1969000000 3.9249600000 19.9397400000
CARBON 6.0 6.8280600000 3.7834200000 21.1913200000
CARBON 6.0 5.7697600000 7.6933500000 17.4241800000
CARBON 6.0 7.2043100000 7.9413600000 17.8281100000
CARBON 6.0 5.5051400000 7.0409700000 14.5903800000
CARBON 6.0 6.8905200000 6.9194700000 14.7626200000
CARBON 6.0 7.7396400000 7.5379800000 13.8285700000
HYDROGEN 1.0 8.8190700000 7.4520600000 13.9252200000
CARBON 6.0 7.2169400000 8.2960300000 12.7704100000
HYDROGEN 1.0 7.8667000000 8.7825100000 12.0575600000
CARBON 6.0 5.8260300000 8.4502300000 12.6467800000
HYDROGEN 1.0 5.4143000000 9.0544300000 11.8493100000
CARBON 6.0 4.9881500000 7.8192300000 13.5528400000
HYDROGEN 1.0 3.9090500000 7.9420000000 13.4583700000
CARBON 6.0 7.1538500000 1.1569600000 13.4143900000
CARBON 6.0 4.4018100000 1.3603900000 13.1919900000
CARBON 6.0 6.4791600000 0.3185500000 12.5353300000
CARBON 6.0 5.0837400000 0.4369500000 12.4084900000
HYDROGEN 1.0 7.0116000000 -0.4099400000 11.9434600000
HYDROGEN 1.0 8.2399000000 1.0702400000 13.4937600000
HYDROGEN 1.0 3.3185600000 1.4368700000 13.0953100000
HYDROGEN 1.0 4.5549800000 -0.1997300000 11.7165200000
CARBON 6.0 6.1105700000 3.9639000000 22.3866100000
CARBON 6.0 4.1216300000 4.4424400000 21.1020100000
HYDROGEN 1.0 7.8732900000 3.5217100000 21.2520500000
CARBON 6.0 4.7606000000 4.2868500000 22.3363800000
HYDROGEN 1.0 6.6064200000 3.8406000000 23.3428500000
HYDROGEN 1.0 4.2065000000 4.4170700000 23.2667100000
HYDROGEN 1.0 3.0674000000 4.6893500000 21.0889000000
HYDROGEN 1.0 7.4249200000 7.7545300000 18.8583200000
HYDROGEN 1.0 7.6651700000 8.9049700000 17.7652100000
HYDROGEN 1.0 5.3324000000 8.6487800000 17.2222700000
HYDROGEN 1.0 5.5015000000 7.1039000000 18.2759400000
$END

=====Effective Core Potential Data=====

The effective core potential (ECP) data is entered after the coordinates. It starts with "$ECP", which must be preceded with a space. The atoms of the molecule are listed in the same order as in the coordinates section and the parameters for the ECP are listed after each atom. Note that for any atom that does NOT have an ECP, one must enter "ECP-NONE" or "NONE" after each atom without an ECP.

$ECP
MO-ECP GEN 28 3
5 ----- f potential -----
-0.0469492 0 537.9667807
-20.2080084 1 147.8982938
-106.2116302 2 45.7358898
-41.8107368 2 13.2911467
-4.2054103 2 4.7059961
3 ----- s-f potential -----
2.8063717 0 110.2991760
44.5162012 1 23.2014645
82.7785227 2 5.3530131
4 ----- p-f potential -----
4.9420876 0 63.2901397
25.8604976 1 23.3315302
132.4708742 2 24.6759423
57.3149794 2 4.6493040
5 ----- d-f potential -----
3.0054591 0 104.4839977
26.3637851 1 66.2307245
183.3849199 2 39.1283176
98.4453068 2 13.1164437
22.4901377 2 3.6280263
S NONE
S NONE
S NONE
S NONE
S NONE
S NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
C NONE
H NONE
C NONE
H NONE
C NONE
H NONE
C NONE
H NONE
C NONE
C NONE
C NONE
C NONE
H NONE
H NONE
H NONE
H NONE
C NONE
C NONE
H NONE
C NONE
H NONE
H NONE
H NONE
H NONE
H NONE
H NONE
H NONE
$END

-- [[User:M.Zimmer-De Iuliis|mzd]] 16 November 2009

====Using an External File to Define Basis Set in GAMESS(US)====

Since GAMESS(US) has a limited number of built-in ECPs and basis sets, one may want to make GAMESS(US) read an external file that contains the basis set information ECP data using the "EXTFIL" keyword in the $GBASIS command line of the input file. For many metal containing compounds, it is very convenient and time saving to use an effective core potential (ECP) for the core metal electrons, as they are usually not important to the reactivity of the complex or the geometry around the metal. In addition, to make GAMESS(US) use this external file, one must copy the "rungms" file and modify it accordingly. The following is a list of instructions with commands that will work from a terminal. One could also use WinSCP to do all of this with a GUI rather than a TUI.

=====Modifiying rungms to Use Custom Basis Set File=====
1. Copy "rungms" from /scinet/gpc/Applications/gamess to one's own /scratch/$USER/ directory:
cp /scinet/gpc/Applications/gamess/rungms /scratch/$USER/

2. Change to the scratch directory and check to see if "rungms" has copied successfully.
cd /scratch/$USER
ls

3. Edit line 147 of the script.
vi rungms
Move the cursor down to line 147 using the arrow keys. It should say "setenv EXTBAS /dev/null". Using the arrow keys, move the cursor to the first "/" and then hit "i" to insert text. Put the path to your external basis file here. For example, /scratch/$USER/basisset. Then hit "escape". To save the changes and exit vi, type ":" and you should see a colon appear at the bottom of the window. Type "wq" (which should appear at the bottom of the window next to the colon) and then hit enter. Now you are done with vi.

=====Creating a Custom Basis Set File=====
1. To create a custom basis set file, you need create a new text document. Our group's common practice is to comment out the first line of this file by inserting an exclamation mark (!) followed by noting the specific basis sets and ECPs that are going to be used for each of the atoms. Let us use the molecule Mo(CO)6, Molybdenum hexacarbonyl, as an example. Below is the first line of the the external file, which we will call "CUSTOMMO" (NOTE: you can use any name for the external file that suits you, as long as it has no spaces and is 8 characters or less).

! 6-31G on C and O and LANL2D2 ECP on Mo

2. The next step is to visit the [https://bse.pnl.gov/bse/portal EMSL Basis Set exchange] and select C and O from the periodic table. Then, on the left of the page, select "6-31G" as the basis set. Finally, make sure the output is in GAMESS(US) format using the drop-down menu and then click "get basis set".

[[File:C_O_6_31G_basisset.JPG|centre]]

3. A new window should appear with text in it. For our example case, the text looks like this:

! 6-31G EMSL Basis Set Exchange Library 10/13/09 11:12 AM
! Elements References
! -------- ----------
! H - He: W.J. Hehre, R. Ditchfield and J.A. Pople, J. Chem. Phys. 56,
! Li - Ne: 2257 (1972). Note: Li and B come from J.D. Dill and J.A.
! Pople, J. Chem. Phys. 62, 2921 (1975).
! Na - Ar: M.M. Francl, W.J. Petro, W.J. Hehre, J.S. Binkley, M.S. Gordon,
! D.J. DeFrees and J.A. Pople, J. Chem. Phys. 77, 3654 (1982)
! K - Zn: V. Rassolov, J.A. Pople, M. Ratner and T.L. Windus, J. Chem. Phys.
! 109, 1223 (1998)
! Note: He and Ne are unpublished basis sets taken from the Gaussian
! program
!
$DATA 
CARBON
S 6
1 3047.5249000 0.0018347
2 457.3695100 0.0140373
3 103.9486900 0.0688426
4 29.2101550 0.2321844
5 9.2866630 0.4679413
6 3.1639270 0.3623120
L 3
1 7.8682724 -0.1193324 0.0689991
2 1.8812885 -0.1608542 0.3164240
3 0.5442493 1.1434564 0.7443083
L 1
1 0.1687144 1.0000000 1.0000000 
OXYGEN
S 6
1 5484.6717000 0.0018311
2 825.2349500 0.0139501
3 188.0469600 0.0684451
4 52.9645000 0.2327143
5 16.8975700 0.4701930
6 5.7996353 0.3585209
L 3
1 15.5396160 -0.1107775 0.0708743
2 3.5999336 -0.1480263 0.3397528
3 1.0137618 1.1307670 0.7271586
L 1
1 0.2700058 1.0000000 1.0000000
$END

3. Now, copy and paste the text between the $DATA and $END headings onto our external text file, CUSTOMMO. We also need to change the change the name of each element to the corresponding symbol in the periodic table. Finally, we need to add the name of the external file next to the element symbol, separated by one space. Note that there should be a blank line separating the basis set information and the first, commented-out line (The line starting with the '!'). The CUSTOMMO should look like this:

! 6-31G on C and O and LANL2D2 ECP on Mo 
C CUSTOMMO
S 6
1 3047.5249000 0.0018347
2 457.3695100 0.0140373
3 103.9486900 0.0688426
4 29.2101550 0.2321844
5 9.2866630 0.4679413
6 3.1639270 0.3623120
L 3
1 7.8682724 -0.1193324 0.0689991
2 1.8812885 -0.1608542 0.3164240
3 0.5442493 1.1434564 0.7443083
L 1
1 0.1687144 1.0000000 1.0000000 
O CUSTOMMO
S 6
1 5484.6717000 0.0018311
2 825.2349500 0.0139501
3 188.0469600 0.0684451
4 52.9645000 0.2327143
5 16.8975700 0.4701930
6 5.7996353 0.3585209
L 3
1 15.5396160 -0.1107775 0.0708743
2 3.5999336 -0.1480263 0.3397528
3 1.0137618 1.1307670 0.7271586
L 1
1 0.2700058 1.0000000 1.0000000

4. Repeat Step 3 above but choose Mo and select the LANL2DZ ECP instead. A new window will pop up with the basis set information as well as the ECP data we need, since we specified the LANL2DZ '''ECP'''. The ECP data is not inserted into the external file, rather it is placed into the input file itself (More on this later).

[[File:Mo_LANL2DZ_basisset.JPG|centre]]

5. After copying the molybdenum basis set information, your fiished external basis set file should look like this:
! 6-31G on C and O and LANL2D2 ECP on Mo 
C CUSTOMMO
S 6
1 3047.5249000 0.0018347
2 457.3695100 0.0140373
3 103.9486900 0.0688426
4 29.2101550 0.2321844
5 9.2866630 0.4679413
6 3.1639270 0.3623120
L 3
1 7.8682724 -0.1193324 0.0689991
2 1.8812885 -0.1608542 0.3164240
3 0.5442493 1.1434564 0.7443083
L 1
1 0.1687144 1.0000000 1.0000000 
O CUSTOMMO
S 6
1 5484.6717000 0.0018311
2 825.2349500 0.0139501
3 188.0469600 0.0684451
4 52.9645000 0.2327143
5 16.8975700 0.4701930
6 5.7996353 0.3585209
L 3
1 15.5396160 -0.1107775 0.0708743
2 3.5999336 -0.1480263 0.3397528
3 1.0137618 1.1307670 0.7271586
L 1
1 0.2700058 1.0000000 1.0000000 
Mo CUSTOMO
S 3
1 2.3610000 -0.9121760
2 1.3090000 1.1477453
3 0.4500000 0.6097109
S 4
1 2.3610000 0.8139259
2 1.3090000 -1.1360084
3 0.4500000 -1.1611592
4 0.1681000 1.0064786
S 1
1 0.0423000 1.0000000
P 3
1 4.8950000 -0.0908258
2 1.0440000 0.7042899
3 0.3877000 0.3973179
P 2
1 0.4995000 -0.1081945
2 0.0780000 1.0368093
P 1
1 0.0247000 1.0000000
D 3
1 2.9930000 0.0527063
2 1.0630000 0.5003907
3 0.3721000 0.5794024
D 1
1 0.1178000 1.0000000

-- [[User:M.Zimmer-De Iuliis|mzd]] 21 September 2010

====A Modified BASH Script for Runnning GAMESS(US)====
Below please find the bash script that we use to run GAMESS(US) on a single node with 8 processors.

One quirk of GAMESS(US) is that it will NOT write over old or failed jobs that have the same name as the input file you are submitting. For example: my input file name is "mo_opt.inp" and I submit this job to the queue. However, it comes back seconds later with an error. The log file says that I have typed an incorrect keyword, and lo and behold, I have a comma where it shouldn't be. Such typos can be common. If you simply try to re-submit, GAMESS(US) will fail again, because it has written a .log file and some other files to the /scratch/user/gamess-scratch/ directory. These files must all be deleted before you re-submit your fixed input file.

This script takes care of this annoying problem by deleting failed jobs with the same file name for you.

Here it is:

#!/bin/bash
#PBS -l nodes=1:ppn=8,walltime=48:00:00

## To submit type: qsub x.sh

# If not an interactive job (i.e. -I), then cd into the directory where
# I typed qsub.
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then
if [ -n "$PBS_O_WORKDIR" ]; then
cd $PBS_O_WORKDIR
fi
fi

# the input file is typically named something like "gamesjob.inp"
# so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"

find /scratch/user/gamess-scratch -type f -name ${NAME:-safety_net}\* -exec /bin/rm {} \;

# loading modules should be done in the submission script.
# It is not recommended by SciNet staff to put this command
# in the .bashrc file.
module load gamess

# run the program

/scratch/user/rungms $NAME 00 8 8 >& $NAME.log

====A Script to Add the $VIB Group for Hessian Restarts in GAMESS(US)====

Sometimes, a optimization + vibrational analysis or just a plain vibrational analysis must be restarted. This can be because the two day time limit has been exceeded or perhaps there was an error during calculation. In any case, when this happens, the job must be restarted. In GAMESS(US), you can restart a vibrational analysis from a previous one and it will utilize the frequencies that were already computed in the failed run.

For example, if one submits the input file "job_name.inp" and it fails before it has finished, then one must utilize the file "job_name.rst", which contains data that is required to restart the calculation. This file is located in the /scratch/user/gamess-scratch directory. Data from the "job_name.rst" file must be appended at the end of the new input file (after the coordinates and ECP section if it is present) to restart the calculation, letus call it "job_name_restart.inp"

A shortened version of the "job_name.rst" file looks like this:

ENERGY/GRADIENT/DIPOLE RESTART DATA FOR RUNTYP=HESSIAN
job_name
$VIB
IVIB= 0 IATOM= 0 ICOORD= 0 E= -3717.1435124522
-5.165258381E-04 1.584665821E-02-1.206270555E-02-2.241461728E-03 3.176050715E-03
-5.706738823E-04 2.502034151E-03 5.130112290E-04-2.716945939E-03 1.357008279E-03
-1.059915305E-03 1.693526456E-03-2.957638907E-04-5.994938737E-04 9.684054361E-04
.
.
.
.

The text eventually ends with one blank line. The $VIB heading and all of the text after $VIB must be appended to the end of file "job_name_restart.inp" and then " $END" must be inserted at the very end of the file.

One could do this, one could cut cut and paste in a text editor, but we have written a small script that will do this automatically. We call it ".vib.sh" but you can call it whatever you like. Here it is:

#!/bin/bash
# script to add vibrational data for a hessian restart

awk '/\$VIB/{p=1}p;END{print " $END"}' /scratch/user/gamess-scratch/$NAME1.rst >> $NAME2.inp

To use it, simply copy it into a new text file with the extension ".sh" and make it executable. Also, you will need to edit the location of the "/scratch/user/gamess-scratch/ directory to match your user name. The two variables in the script, NAME1 and NAME2, represent the name of your ".rst" file and your new ".inp" file, respectively. In the example above, NAME1=job_name (that is, the same name as the .rst file that contains the $VIB data and that was created in the /gamess-scrsatch/ directory) and NAME2=job_name_restart (that is, the name of the new input file that you have prepared and want to copy the $VIB data into).

To run it on a gpc node without submitting it to the job queue, type:

NAME1=job_name NAME2=job_name_restart ./vib.sh

To run it in the queue, type:

qsub vib.sh -v NAME1=job_name,NAME2=job_name_restart

-special thanks to Ramses for help with this

-- [[User:M.Zimmer-De Iuliis|mzd]] 30 September 2010

====Most Commonly Used Headers in The Fekl Lab====

After about a year of using GAMESS(US), we have found that we are most often doing optimizations, frequency analyses, transition state searches and IRC calculations using DFT methods. Here are the input decks thatwe found have worked well for inorganic and organometallic compounds.

=====Optimization Plus Frequency (for a neutral, singlet)=====

$CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1
ECP=READ $END
$SYSTEM TIMLIM=2800 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END
$SCF DIRSCF=.TRUE. FDIFF=.f. DIIS=.T. SOSCF=.F. DAMP=.T. $END
$STATPT OPTTOL=0.00001 NSTEP=500 HSSEND=.t. $END
$FORCE TEMP=298.15 PURIFY=.t. PROJCT=.t. $END
$DATA

=====Frequency Only (for a neutral, singlet)=====

$CONTRL SCFTYP=RHF RUNTYP=HESSIAN DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1
ECP=READ $END
$SYSTEM TIMLIM=2800 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END
$SCF DIRSCF=.TRUE. FDIFF=.f. DIIS=.T. SOSCF=.F. DAMP=.T. $END
$FORCE METHOD=SEMINUM VIBANL=.TRUE. PROJCT=.T. PURIFY=.T. $END
$DATA

=====Transition State Search (for a neutral, singlet)=====

$CONTRL SCFTYP=RHF RUNTYP=SADPOINT DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1
ECP=READ $END
$SYSTEM TIMLIM=2850 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END
$SCF DIRSCF=.TRUE. FDIFF=.f. DIIS=.T. SOSCF=.F. $END
$STATPT STSTEP=0.05 OPTTOL=0.00001 NSTEP=500 HESS=CALC HSSEND=.t.
STPT=.FALSE. $END
$FORCE METHOD=SEMINUM VIBANL=.TRUE. PURIFY=.T. PROJCT=.T. $END
$DATA

=====IRC (Intrinsic Reaction Coordinate following forward reaction) Calculation (for a neutral, singlet)=====

$CONTRL SCFTYP=RHF RUNTYP=IRC DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1
ECP=READ $END
$IRC OPTTOL=0.00001 STRIDE=0.05 NPOINT=5000 SADDLE=.TRUE. FORWRD=.F.
$END
$SYSTEM TIMLIM=2850 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END
$SCF DIRSCF=.TRUE. FDIFF=.f. $END
$FORCE TEMP=298.15 PURIFY=.t. PROJCT=.t. $END
$DATA

-- [[User:M.Zimmer-De Iuliis|mzd]] 21 September 2010

====How to Run an IRC Calculation Using GAMESS(US)====

An IRC or Intrinsic Reaction Coordinate calculation follows the imaginary mode of the vibrational analysis of a transition state calculation. In GAMESS(US), you can choose to follow the forward (towards the products) or backward (toward the reactants) direction. As shown above in the IRC header that we use, the direction of the IRC calculation is controlled by the "FORWRD" key word. Using "FORWRD=.T." means that the IRC is following the forward direction, while using "FORWRD=.F." means that the IRC calculation is following the backward direction.

Let us say we want to perform an IRC. In order to perform an IRC calculation, you must first perform a vibrational analysis of you molecule and check to ensure there is only 1 negative frequency. If that is the case, then the vibrational analysis completed successfully and there will be a file, let us call it "job_name.dat" in the "/users/$USER/gamess-scratch/" directory (where $USER is your user name) with the extension ".dat". In this file is data that is required for the IRC input file.

To prepare your IRC input file, prepare an input file using the coordinates of the optimized structure of the transition state. This can be from ChemCraft or Avogadro or MacMolPlt - what ever you prefer to use. Then copy and paste the IRC header above or use your own parameters. Call it whatever you want, as long as it has an ".inp" extension. Let us call in "irc_job.inp".

For example, the "STRIDE" value determines the "size" of the steps between each point on the IRC graph. If you increase the value of the stride, say from 0.05 to 0.1, then the steps in between each point become larger and you will approach the minimum faster (this will give you fewer data points should you chose to plot the IRC data). Decreasing the stride value, say from 0.05 to 0.01 will make the steps in between each point become smaller and you may not reach the minimum of the reaction coordinate in the alloted time period.

You should now have an input file with an IRC header, the coordinates of the transition state and basis set and ECP information called "irc_job.inp".

Now you need to use the "job_name.dat" file in the "/users/$USER/gamess-scratch/" In this file are a number of blocks of data that are sandwiched between a line that contains only " $HESS" and a line that contains only " $END". What you need is the LAST of these blocks of text and it has to be copied and pasted directly below the last entry of your input file.

This can be difficult and time consuming, as the .dat files can be very large (sometimes over 150 MB) and cumbersome to navigate through. However, we have written a script, similar to the .vib.sh script, that can help you out with this. Basically, this script does all the copying and pasting for you.

Here it is:
#!/bin/bash
# script to add hessian data for an IRC calculation

awk '/\$HESS/{arr="";f=1} f {arr=(arr)?arr ORS $0:$0} /\$END/{f=0} END {print arr}' /scratch/$USER/gamess-scratch/$DAT.dat >> $IN.inp

To use it, simply copy it into a new text file with the name "irc.sh" and make it executable. Also, you will need to edit the location of the "/scratch/user/gamess-scratch/ directory to match your user name. The two variables in the script, $DAT and $IN, represent the name of your ".dat" file and your new ".inp" file, respectively. Using our current example, $DAT=job_name and In the example above, $IN=irc_job (that is, the same name as the .dat file that contains the $HESS data and that was created in the /gamess-scrsatch/ directory) and IN=irc_job (that is, the name of the new input file that you have prepared and want to copy the $HESS data into).

To run it on a gpc node without submitting it to the job queue, type:

DAT=job_name IN=irc_job ./irc.sh

To run it in the queue, type:

qsub irc.sh -v DAT=job_name,IN=irc_job

-- [[User:M.Zimmer-De Iuliis|mzd]] 21 October 2010

===Vienna Ab-initio Simulation Package (VASP)===
Please refer to the [http://www.vasp.at VASP page].

User-supplied content below.

====Tips from the Polanyi Lab====
Using VASP on SciNet

Logon using SSH
login.scinet.utoronto.ca

then ssh to the TCS cluster
ssh tcs01

change directory to
cd /scratch/imcnab/test/Si111 - or whatever other directory is convenient.

VASP is contained in the directory imcnab/bin

To submit a job, first edit (at least) the POSCAR file and other VASP
input files as necessary.

=====Input Files=====
The minimum set of input files is:

'''vasp.script''' - script file telling TCS to run a VASP job - must be edited to run in current working directory.

'''POSCAR''' - specifies supercell geometry and "ionic" positions (i.e. atomic centres) and whether relaxation allowed. Ionic positions may be given in cartesion coordinates (x,y,z in A) or "absolute", which are fractions of the unit cell vectors. CONTCAR is always in absolute coords, so after the first run of any job, you'll find yourself running in absolute coords. VMD can be used to change these back to cartesian coordinates.

'''INCAR''' - specifies parameters to run the job. INCAR is free format - can put input commands in ANY order.

'''POTCAR''' - specifies the potentials to use for each atomic type. Must be in the same order as the atoms are first met in POSCAR

'''KPOINTS''' - specifies the number and position of K-points to use in the calculation.

Any change of name or directory needs to be edited into the job script. The job script name is "vasp.script".

VASP attempts to read initial wavefunctions from WAVECAR, so if a job is run in steps, leaving the WAVECAR file on the working directory is an efficient way to start the next stage of the calculation

VASP also writes CONTCAR which is of the same format as POSCAR, and can simply be renamed if it is to be used as the starting point for a new job.

Submit the job to load-leveller with the command llsubmit ./vasp.script from the correct working directory.

can check the status of a job with llq

can cancel a job using llcancel tcs-fXXnYY.$PID where tcs number etc is shown by llq

==
INPUT FILES ==

The minimum set of input files is:

'''vasp.script''' - script file telling TCS to run a VASP job - must be edited to run in current working directory.

'''POSCAR''' - specifies supercell geometry and "ionic" positions (i.e. atomic centres) and whether relaxation allowed. Ionic positions may be given in cartesion coordinates (x,y,z in A) or "absolute", which are fractions of the unit cell vectors. CONTCAR is always in absolute coords, so after the first run of any job, you'll find yourself running in absolute coords. VMD can be used to change these back to cartesian coordinates.

'''INCAR''' - specifies parameters to run the job. INCAR is free format - can put input commands in ANY order.

'''POTCAR''' - specifies the potentials to use for each atomic type. Must be in the same order as the atoms are first met in POSCAR

'''KPOINTS''' - specifies the number and position of K-points to use in the calculation.

Any change of name or directory needs to be edited into the job script.
The job script name is "'''vasp.script'''".

VASP attempts to read initial wavefunctions from WAVECAR, so if a job is
run in steps, leaving the WAVECAR file on the working directory is an
efficient way to start the next stage of the calculation

VASP also writes CONTCAR which is of the same format as POSCAR, and can
simply be renamed if it is to be used as the starting point for a new job.

Submit the job to load-leveller with the command
llsubmit ./vasp.script from the correct working directory.

can check the status of a job with
llq

can cancel a job using
llcancel tcs-fXXnYY.$PID where tcs number etc is shown by llq

===== GENERAL NOTES =====

MUCH faster to use ISPIN=1, no-spin (corresponds to RHF, rather than
ISPIN=2 which corresponds to URHF). So far, I've not found a system where the atom positions differ, or where the calculated electronic energy differs by more than 1E-4, which is the convergence
criteria set.

MUCH faster to use real space LREAL = A, NSIM=4.

So, ''always'' optimize in real space first, then re-optimize in reciprocal space. This does NOT guarantee, a one-step optimization in reciprocal space. May still need to progressively
relax a large system.

'''Relaxing a large system.'''
If you attempt to relax a large system in one step, it will usually fail.

The starting geometry is usually an unrelaxed molecule above an unrelaxed surface.
The bottom plane of the surface will NEVER be relaxed, because this corresponds to the fixed boundary condition of REALITY.

First, relax the molecule alone (assuming you have already found a good starting position from single point calcultions, place the molecule closer to the surface than you think it should be (say 0.9 VdW radii away).

Then ALSO allow the top layer of the surface to relax.
Then ALSO allow the second top layer of the surface to relax... etc... etc.

If this DOESN'T WORK: Then relax X,Y and Z separately in iterations.
Example. For the following problem, representing layers of the crystal going DOWN from the top (Z pointing to the top of the screen)

Molecule 
Layer 1 
Layer 2 
Layer 3 
Layer 4 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

we can try the following relaxation schemes: 

Successive relaxation, Layer by Layer: 
(1) 
Molecule XYZ Relax 
Layer 1 XYZ fixed 
Layer 2 XYZ fixed 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer. 
Layer 6 - Valence H's, fixed layer 

(2) 
Molecule XYZ Relax 
Layer 1 XYZ Relax 
Layer 2 XYZ fixed 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer. 
Layer 6 - Valence H's, fixed layer 

(3) 
Molecule XYZ Relax 
Layer 1 XYZ Relax 
Layer 2 XYZ Relax 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer. 
Layer 6 - Valence H's, fixed layer 

etc. etc... if this works then you're fine. However, it can happen that even by Layer 2, you're running into real problems, and the ionic relaxation NEVER converges. In which case, I have found the following scheme (and variations thereof) useful:

(1) 
Molecule XYZ Relax 
Layer 1 XYZ fixed 
Layer 2 XYZ fixed 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

(2) 
Molecule XYZ Relax 
Layer 1 XYZ Relax 
Layer 2 XYZ fixed 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

(3) 
Molecule XYZ Relax 
Layer 1 XYZ Relax 
Layer 2 XYZ Relax 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

IF (3) DOESN'T converge THEN TRY

(2') 
Molecule Z Relax, XY FIXED 
Layer 1 Z Relax, XY FIXED 
Layer 2 XYZ Relax 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

- you are allowing the top layers to move only UP or DOWN, while allowing the intermediate
layer 2 to fully relax (actually, there is no way of telling VASP to move ALL atoms by the SAME deltaZ, but that appears to be the effect.
Followed by

(2") 
Molecule XYZ Relax 
Layer 1 XYZ Relax 
Layer 2 XYZ Relax 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

If (2") doesn't work, you need to go back to the output of (2') and vary the cycle - perhaps something like:
(2"') 
Molecule XYZ Relax 
Layer 1 XYZ Relax 
Layer 2 XYZ fixed 
Layer 3 XYZ fixed 
Layer 4 XYZ fixed 
Layer 5 - fixed layer 
Layer 6 - Valence H's, fixed layer 

then try (2") again.

Repeat as necessary. This scheme does appear to work quite well for big unit cells. It can be very difficult to relax as many layers as necessary in a big unit cell.

Experience on the One Per Corner Hole problem shows that it may be necessary to have a large number of UNRELAXED (i.e. BULK silicon) layers underneath the relaxed layers in order to get physically meaningful answers. This is because silicon is so elastic.

===== Problems and solutions: =====

If getting ZBRENT errors, try changing ALGO. Usually use ALGO = Fast, change to ALGO = Normal. With ALGO = Normal, NFREE now DOES correspond to degrees of freedom (maximum suggested setting is 20). Haven't found this terribly helpful.

Many calculations seem to fail after 20 or 30 ionic steps. I suspect a memory leak.

Sometimes the calculation appears to lose WAVECAR... this is not a disaster, just means a slight increase in start time as the first wavefunction is calculated.

If calculation does not finish nicely, can force a WAVECAR generation by doing a purely electronic calculation (these are pretty fast).

VASP is VERY slow at relaxing molecules at surfaces. This is because it doesn't know a molecule is a connected entity. It treats every atom independently.

THEREFORE, MUCH MUCH faster to try molecular positions by hand first.
Do some sample calculations at a few geometries to find a good starting point.

ALSO, once you think you know where the molecule is to be placed, put it too close to the surface, and let it relax outwards... the forces close to the surface are repulsive, and much steeper, so relaxation is FASTER in this direction.

=='''Climate Modelling'''==

The Community Earth System Model (CESM) is a fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states.

Development of a comprehensive CESM that accurately represents the principal components of the climate system and their couplings requires both wide intellectual participation and computing capabilities beyond those available to most U.S. institutions. The CESM, therefore, must include an improved framework for coupling existing and future component models developed at multiple institutions, to permit rapid exploration of alternate formulations. This framework must be amenable to components of varying complexity and at varying resolutions, in accordance with a balance of scientific needs and resource demands. In particular, the CESM must accommodate an active program of simulations and evaluations, using an evolving model to address scientific issues and problems of national and international policy interest.

User guides and information on each version of the model can be found at the following links:

CCSM3: http://www.cesm.ucar.edu/models/ccsm3.0/
CCSM4: http://www.cesm.ucar.edu/models/ccsm4.0/
CESM1: http://www.cesm.ucar.edu/models/cesm1.0/

Please see:

===[[Installing CCSM3]]===

===[[Running CCSM3]]===

===[[Installing CCSM4]]===

===[[Running CCSM4]]===

===[[Post Processing CCSM Output]]===

===[[CCSM4/CESM1 TCS Simulation List]]===

==Engineering==

===ANSYS===

Please refer to the [[Ansys]] page.

==Medicine/Bio==

==High Energy Physics==

==Structural Biology==
Molecular simulation of proteins, lipids, carbohydrates, and other biologically relevant molecules.
===Molecular Dynamics (MD) simulation===
====GROMACS====
Please refer to the [[gromacs|GROMACS]] page
====AMBER====
Please refer to the [[amber|AMBER]] page
====NAMD====
'''Note: the instructions below are rather old, and should not be needed by most users. A version of namd is available on the GPC in the module <tt>namd</tt>. For current information see the [[Namd]] page'''

NAMD is one of the better scaling MD packages out there. With sufficiently large systems, it is able to scale to hundreds or thousands of cores on Scinet. Below are details for compiling and running NAMD on Scinet.

More information regarding performance and different compile options coming soon...

=====Compiling NAMD for GPC=====
Ensure the proper compiler/mpi modules are loaded.
<source lang="bash">
module load intel
module load openmpi/1.3.3-intel-v11.0-ofed
</source>

'''Compile Charm++ and NAMD'''
<source lang="bash">
#Unpack source files and get required support libraries
tar -xzf NAMD_2.7b1_Source.tar.gz
cd NAMD_2.7b1_Source
tar -xf charm-6.1.tar
wget http://www.ks.uiuc.edu/Research/namd/libraries/fftw-linux-x86_64.tar.gz
wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl-linux-x86_64.tar.gz
tar -xzf fftw-linux-x86_64.tar.gz; mv linux-x86_64 fftw
tar -xzf tcl-linux-x86_64.tar.gz; mv linux-x86_64 tcl
#Compile Charm++
cd charm-6.1
./build charm++ mpi-linux-x86_64 icc --basedir /scinet/gpc/mpi/openmpi/1.3.3-intel-v11.0-ofed/ --no-shared -O -DCMK_OPTIMIZE=1
cd ..
#Compile NAMD.
#Edit arch/Linux-x86_64-icc.arch and add "-lmpi" to the end of the CXXOPTS and COPTS line.
#Make a builds directory if you want different versions of NAMD compiled at the same time.
mkdir builds
./config builds/Linux-x86_64-icc --charm-arch mpi-linux-x86_64-icc
cd builds/Linux-x86_64-icc/
make -j4 namd2 # Adjust value of j as desired to specify number of simultaneous make targets.
</source>
--[[User:Cmadill|Cmadill]] 16:18, 27 August 2009 (UTC)

=====Running Fortran=====
On the development nodes, there is an old gcc. The associated libraries are not on the compute nodes. Ensure the line:

module load gcc

is in your .bashrc file.

====LAMMPS====
[[Image:StrongScalingLAMMPS.png|thumb|320px|right|Strong scaling test on GPC with OpenMPI and IntelMPI on Ethernet and InfiniBand]]
[[Image:WeakScalingLAMMPS.png|thumb|320px|right|Weak scaling test on GPC with OpenMPI and IntelMPI on Ethernet and InfiniBand]]
LAMMPS is a parallel MD code that can be found [http://lammps.sandia.gov/ here].

'''Compiling LAMMPS on GPC'''

First, make sure the ''intel/intel-v11.1.072'' and ''openmpi/1.4.1-intel-v11.0-ofed'' modules are loaded.

To download LAMMPS into your directory, instructions can be found [http://lammps.sandia.gov/download.html here].

In ''src/MAKE/Makefile.openmpi'', make the following changes:

FFT_INC = -DFFT_NONE

<nowiki># FFT_PATH =</nowiki>

<nowiki># FFT_LIB =</nowiki>

Note that the FFT library is required for using PPPM from the KSPACE package.

In the ''src'' directory, run ''make openmpi'', which should produce a ''lmp_openmpi'' executable.

--[[User:sellan|sellan]] 14:08 Jul 28 2011

'''Scaling Tests on GPC'''

Results from strong scaling tests for LAMMPS using EAM potentials on GPC are shown in the graph on the right. Test simulation ran 500 timesteps for 4,000,000 atoms.

Results from weak scaling tests for LAMMPS using EAM potentials on GPC are shown in the graph on the right. Test simulation ran 500 timesteps for 32,000 atoms per processor.

OpenMPI version used: openmpi/1.4.1-intel-v11.0-ofed

IntelMPI version used: intelmpi/impi-4.0.0.013

LAMMPS version used: 15 Jan 2010

'''Summary of Scaling Tests'''

Results show good scaling for both OpenMPI and IntelMPI on Ethernet up to 16 processors, after which performance begins to suffer. On Infiniband, excellent scaling is maintained to 512 processors.

IntelMPI shows slightly better performance compared to OpenMPI when running with Infiniband.

--[[User:jchu|jchu]] 14:08 Feb 2, 2010

===Monte Carlo (MC) simulation===

Oldwiki.scinet.utoronto.ca:System Alerts

2016-11-07T17:50:46Z

Nolta: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|down|link=P8]][[P8]]
|[[File:up.png|down|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|}

Mon Nov 7 11:59:00 EST 2016 File system has been restored. Jobs are being scheduled again. Please resubmit jobs if they crashed or had issues last night or this morning.

Mon Nov 7 10:40:00 EST 2016 Due to this issue, many jobs will have either crashed, or have not had a change to write their output; please check any jobs you had running overnight. The scratch file system is expected to be back up soon.

Mon Nov 7 9:40:00 EST 2016 Scratch file system filled up overnight. We are investigating how to mitigate this. In the meantime, the job scheduler has been stopped, so no new jobs will start (but will remain in the queue).

Mon Nov 7 8:00:00 EST 2016 Apparent file system issues.

Fri Oct 28 23:00:00 EDT 2016 The login nodes and devel nodes of the GPC, P7 and BGQ, as well as the datamover nodes, will be rebooted between 2 am and 6 am on Sat Oct 29. Running and queued jobs will not be affected, but interactive sessions will be closed.

Mon Sep 26 10:33:47 EDT 2016 HPSS schedule is back to normal operations.

Sun Sep 25 12:37:12 EDT 2016 Problems resolved. Systems have started coming online. Check the status "lights" above.

Sun Sep 25 10:16:37 EDT 2016 Power outage tripped main breaker and other circuits. Power has been restored to site but there may be an issue with cooling system power that needs to be resolved before any compute systems can be restarted

Sun Sep 25 09:28:15 EDT 2016 Staff enroute to site. After assessing situation will give ETA for recovery.

Sun Sep 25 08:46 EDT 2016 Power outage at datacentre.

GPC Quickstart

2016-10-26T20:52:07Z

Nolta:

{{Infobox Computer
|image=[[Image:University_of_Tor_79284gm-a.jpg|center|300px|thumb]]
|name=General Purpose Cluster (GPC)
|installed=June 2009
|operatingsystem= Linux Centos 6.4
|loginnode= gpc01..gpc08 (from <tt>login.scinet</tt>)
|nnodes=3864 (30,912 cores)
|rampernode=16 Gb
|corespernode=8 (16 threads)
|interconnect=840 nodes 1:1 DDR, 3024 nodes 5:1 QDR
|vendorcompilers=icc (C) ifort (fortran) icpc (C++)
|queuetype=[[Moab | Moab/Torque]]
}}

__TOC__

===Specifications===
The General Purpose Cluster is an extremely large cluster (ranked [http://www.top500.org/list/2009/06/100 16th] in the world at its inception, currently still at #391) and is where most computations are to be done at SciNet.
The GPC consists of 3,780 nodes with 16 GB of memory or more and 8 Intel cores each. The nodes use a shared parallel file system and have no local disks. The compute nodes are accessed through a queuing system that allows jobs with a minimum of 15 minutes and maximum wall time of 48 hours .

Technical details: The GPC is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the [http://www.hpcwire.com/features/HPC-Vendors-Jump-On-Nehalem-42360237.html first in the world] to make use of the new chips). The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,912 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking DDR InfiniBand while the rest of the nodes are connected with 5:1 blocked QDR InfiniBand.

Note: from an operational standpoint, the contributed systems [[Gravity]] and [[Sandy]], as well as the [[Visualization_Nodes|Viz]] system, are more or less part of the GPC.

===Login===

First login via [[Ssh | ssh]] with your SciNet account at <tt>login.scinet.utoronto.ca</tt>. This puts you on a 'login node', but not yet on the GPC. From the login node, you have to ssh to one of the 8 development nodes (<tt>gpc01</tt>, <tt>gpc02</tt>, ..., <tt>gpc08</tt>) to compile/test your code.

===Compile/Devel Nodes===

To get to one of the devel nodes from a scinet login node, you can ssh to <tt>gpc01</tt>..<tt>gpc08</tt>. You may also just use the <tt>gpc</tt> command to take directly to the dev node with lowest cpu load (reassessed every 5 minutes).

Except for these four development nodes on the GPC, all other nodes are 'compute nodes' that can be used only through the scheduler. The devel nodes have the same hardware configuration as most of the compute nodes except with more memory -- 8 processing cores with 36 GB RAM and QDR Infiniband. You can compile and test your codes on these devel nodes. To interactively test on more than 8 processors, you can submit an [[GPC_Quickstart#Submitting_an_Interactive_.28Debug.29_Job | interactive job request]] to get time-limited command-line access to a compute node.

Your [[Storage_Quickstart | home directory]] is in <tt>$HOME</tt>. You have 50 GB there that is backed up. In addition, you have a larger, non-backed up space in <tt>$SCRATCH</tt> for temporary files. Note that your directories are encoded in the environment variables <tt>$HOME</tt> and <tt>$SCRATCH</tt>. Currently <tt>$HOME=/home/G/GROUP/USER</tt> and <tt>$SCRATCH=/scratch/G/GROUP/USER</tt> or <tt>/scratch2/G/GROUP/USER</tt> (where GROUP is your group's name, G is the first letter of that groups name, and USER is your user name). But the locations can change as the storage system evolves and grows with demand, so please use the environment variables!

The GPC devel and compute nodes do not have local disks. Instead, <tt>$HOME</tt> and <tt>$SCRATCH</tt> are shared parallel file systems (the file system is called GPFS), which means that your files are seen on all the nodes.

'''Your home directory cannot be written to by the compute nodes!''' Thus, to run jobs, you'll use the <tt>$SCRATCH</tt> directory (currently <tt>/scratch/g/group/USER</tt> but again, use the environment variable). Here, there is a large amount of disk space, but it is not backed up. Thus it makes sense to keep your codes in /home, compile there, and then run them in the /scratch directory.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including any of the compilers - , you will have to use the `module' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

Note that to use even the gcc compilers you will have to do
<pre>
$ module load gcc
</pre>
but in fact you probably should use the intel compilers installed on this system as they often produce faster executables (and occasionally, much faster.)

A list of the installed software and more information on using the module command is available in [[Software_and_Libraries | Software & Libraries]]. Or, when logged in to the gpc, you can
see all available modules on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load intel
</pre>
To unload a module
<pre>
$ module unload intel
</pre>
To unload all modules
<pre>
$ module purge
</pre>
To list all loaded modules
<pre>
$ module list
</pre>

These commands should go in your submission scripts to make sure you
are using the correct packages. It is possible to load them in your .bashrc files as well, but this is generally not recommended (see [[Important .bashrc guidelines]]), especially if you routinely have to flip back and forth between modules.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' affect other shell environments; in particular, a queued job that is running is unaffected by you interactively loading a module, and conversely you loading a module at the prompt and then submitting a job does not ensure that the module is loaded when the job runs. To ensure that a module is loaded when a job runs, be sure to put your <tt>module load</tt> command in your job submission script.

Again more information on modules, and how to resolve dependencies between modules can be found on the [[Software_and_Libraries | Software & Libraries]] page.

===Compilers===

The intel compilers are icc/icpc/ifort for C/C++/Fortran, and are available with the default module "intel". The intel compilers are recommended over the GNU compilers. Documentation about icpc is available at
http://software.intel.com/en-us/articles/intel-software-technical-documentation/. The Intel compilers accept many of the options that the GNU compilers accept, but tend to produce faster programs on our system. If, for some reason, you really need the GNU compilers, the latest version of the GNU compiler collection (currently 4.4.0) is available by loading the "gcc" module, with gcc/g++/gfortran for C/C++/Fortran. [[Co-array_Fortran_on_the_GPC|Coarray fortran]] is support by the intel compilers from version 12 upwards and by the GNU fortran compiler version 5.2.0. Note that the f77/g77 compilers are not supported, but the available fortran compilers are able to compile fortran 77.

To ensure that the intel compilers are in your <tt>PATH</tt> and their libraries are in your <tt>LD_LIBRARY_PATH</tt>, use the command

<pre>
$ module load intel/15.0.2
</pre>

Optimize your code for the GPC machine using of at least the following compiler flags:
<pre>
-O3 -xHost
</pre>
(or <tt>-O3 -march=native</tt> for the GNU compilers).

*If your program uses openmp, add <tt>-fopenmp</tt> for GNU compilers.
*If you get the warning <tt>feupdatreenv is not implemented</tt>, add -limf to the link line.
*If you need to link in the MKL libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code. '''Note that this give the link line for the command prompt. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''
*More questions about compiling? See the [[FAQ#Compiling_your_Code|FAQ]].

===Debuggers===

* '''ddt''' - Allinea's graphical parallel debugger, in the <tt>ddt</tt> module. Highly recommended!
* '''gdb''' - The GNU Debugger, available in the <tt>gdb</tt> module.
* '''idbc/idb''' - The intel debuggers, part of the <tt>intel</tt> module(s).
* '''ddd''' - A graphical debuggerThe GNU Debugger, available in the <tt>ddd</tt> module.

Note that to debug code, you have to give the <tt>-g</tt> flags to the compiler. The intel compiler needs the additional option <tt>-debug parallel</tt> to debug threaded/OpenMP code.

===[[ GPC_MPI_Versions | MPI]]===

SciNet currently provides multiple MPI libraries for the GPC; [http://www.open-mpi.org/ OpenMPI], and [http://software.intel.com/en-us/intel-mpi-library/ IntelMPI]. We currently recommend OpenMPI as the default, as it quite reliably demonstrates good performance on the infiniband network (and did so too on the ethernet network). For full details and options see the complete [[ GPC_MPI_Versions | '''MPI''']] section.

The MPI libraries are compiled with both the gnu compiler suite and the intel compiler suite. To use (for instance) the intel-compiled OpenMPI libraries, which we recommend as the default (and use for most of our examples here), use

<pre>
$ module load intel/15.0.2 openmpi/intel/1.6.4
</pre>

in your job submission scripts and on the command-line before compiling. Putting these in your <tt>.bashrc</tt> is no longer recommended (since Oct 10, 2013).

Other combinations behave similarly.

The MPI libraries define the wrappers mpicc/mpicxx/mpif90/mpif77 as wrappers around the appropriate compilers, which ensure the appropriate include and library directories are used in the compilation and linking steps.

We currently recommend the Intel + OpenMPI combination. However, if you require the GNU compilers as well as MPI, you will want to find the most recent openmpi module available with `gcc' in the version name. This will enable development and runtime with gcc/g++/gfortran and OpenMPI.


For mixed OpenMP/MPI code using Intel MPI, add the compilation flag -mt_mpi for full thread-safety (no such flag is necessary for OpenMPI).

===Submitting A Batch Job===

The SciNet machines are shared systems, and jobs that are to run on them are submitted to a queue; the
[[Moab | scheduler]] then orders the jobs in order to make the best use of the machine, and has them launched
when resources become availble. The intervention of the scheduler can mean that the jobs aren't
quite run in a first-in first-out order. The scheduler used on the GPC is called Moab (with Torque as the 'resource manager').

The scheduler will have the job run on one or more of the compute nodes of the GPC. There are a few important differences between the devel nodes and the compute nodes:

# As stated above already, on compute nodes, your home directory is read-only. You have to run your jobs from the $SCRATCH directory instead. See [https://support.scinet.utoronto.ca/wiki/index.php/Data_Management Data Management ] for more details on the file systems at SciNet.

# The available memory on compute nodes is approximately 14GB (16GB - 2GB for the operating system). The devel nodes have 36GB, but this is shared with all the users of the node.

# Some libraries, especially those for graphics, are not installed on the compute nodes. This leaves more memory available for your job, but if you have an application that requires such libraries (notably <tt>R</tt> and <tt>octave</tt>), you will need to "<tt>module load extras</tt>" in your job script to make them work.

The maximum [[wallclock time]] for a job in the queue is 48 hours; computations that will take longer than
this must be broken into 48-hour chunks and run as several jobs. Also a minimum job length of 15 minutes is enforced and shorter jobs
should be batched together. The usual way to do this is with [[checkpoints]],
writing out the complete state of the computation every so often in such a way that a job can be restarted from
this state information and continue on from where it left off. Generating [[checkpoints]] is a good idea anyway,
as in the unlikely event of a hardware failure during your run, it allows you to restart without having lost much work.

There are limits to how many jobs you can submit. If your group has a default account, up to 32 nodes at a time for 48 hours per job on the GPC cluster are allowed to be queued. This is a total limit, e.g., you could request 64 nodes for 24 hours. Jobs of users with an LRAC or NRAC allocation will run at a higher priority than others while their resources last. Because of the group-based allocation, it is conceivable that your jobs won't run if your colleagues have already exhausted your group's limits.

Note that scheduling big jobs greatly affects the queue and other users, so you have to talk to us first to run massively parallel jobs (> 2048 cores). We will help make sure that your jobs start and run efficiently.

If your job should run in fewer than 48 hours, specify that in your script -- your job
will start sooner. (It's easier for the [[Moab | scheduler]] to fit in a short job than a long job). On the downside, the
job will be killed automatically by the queue manager software at the end of the specified [[wallclock time]], so if you
guess wrong you might lose some work. So the standard procedure is to estimate how long your job will take and
add 10% or so.

You interact with the queuing system through the queue/resource manager, [[Moab | Moab]] and [[Moab | Torque]]. To see all the jobs in the queue use
<pre>
$ showq
</pre>

To submit your own job, you must write a script which describes the job and how it is to be run (a sample script [[GPC_Quickstart#Submission_Script | follows]]) and submit it to the queue, using the command
<pre>
$ qsub [SCRIPT-FILE-NAME]
</pre>
where you will replace <tt>[SCRIPT-FILE-NAME]</tt> with the file containing the submission script. This will return a job ID, for example 31415, which is used to identify the jobs. Information about a queued job can be found using
<pre>
$ qstat [JOB-ID]
</pre>
and jobs can be canceled with the command
<pre>
$ canceljob [JOB-ID]
</pre>

Again, these commands have many options, which can be read about on their man pages.

Much more information on the queueing system is available on our [[Moab | queue]] page.

====Batch Submission Script: MPI====

A sample submission script is shown below for an mpi job with the <tt> #PBS </tt> directives at the top and the rest being
what will be executed on the compute node.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC
#
#PBS -l nodes=2:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2 openmpi/intel/1.6.4

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 16 ./a.out
</source>

The lines that begin <tt>#PBS</tt> are commands that are parsed and interpreted by qsub at submission time, and control administrative things about your job. In this example, the script above requests two nodes, using 8 processors per node, for a [[wallclock time]] of one hour. (The resources required by the job are listed on the <tt>#PBS -l</tt> line.) Other options can be given in other <tt>#PBS</tt> lines, such as <tt>#PBS -N</tt>, which sets the name of the job.

The rest of the script is run as a bash script at run time. A bash shell on the first node of the two nodes that are requested executes these commands as a normal bash script, just as if you had run this as a shell script from the terminal. The only difference is that PBS sets certain environment variables that you can use in the script. <tt>$PBS_O_WORKDIR</tt> is set to be the directory that the command was 'submitted' from - eg, <tt>$SCRATCH/SOMEDIRECTORY</tt>. The script then uses the <tt>mpirun</tt> command to launch the job.



====Submitting Collections of Serial Jobs====

You cannot run purely serial jobs on the GPC (or any of SciNet's systems), as this would mean only one core out of 8 is used. If you have serial jobs, you have to bunch them together.
SciNet-approved methods for running collections of serial jobs can be found on the [[User_Serial|serial run wiki page]].

====Batch Submission Script: OpenMP====

For running OpenMP jobs, the procedure is similar as for MPI jobs:

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (OpenMP)
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=8
./a.out
</source>

Note that [[Introduction_To_Performance#Throughput | in some circumstances]] it can be more efficient to run (say) two jobs each running on four threads than one job running on eight threads. In that case you can use the same `ampersand-and-wait' technique outlined for serial jobs (see [[User_Serial|serial run wiki page]]) for less-than-eight-core OpenMP jobs.

====Hybrid MPI/OpenMP jobs====

'''Using Intel MPI'''

Here is how to run hybrid codes using intelmpi::

http://software.intel.com/en-us/articles/hybrid-applications-intelmpi-openmp/

Make sure you compile with the -mt_mpi option to the compilers to use the thread safe libraries.
Set the environment variable I_MPI_PIN_DOMAIN:
<pre>
$ export I_MPI_PIN_DOMAIN=omp
</pre>
This will set the process pinning domain size to be equal to OMP_NUM_THREADS (which you should set to the desired number of threads per mpi process). Therefore, each MPI process can create $OMP_NUM_THREADS number of children threads for running within the corresponding domain. If OMP_NUM_THREADS is not set, each node is treated as a separate domain (which will allow as many threads per MPI processes as there are cores).

In addition, when invoking mpirun, you should add the argument "-ppn X", where X is the number of MPI processes per node.
For example:
<pre>
$ mpirun -ppn 2 -np 8 [executable]
</pre>
would start 2 mpi processes of <tt>[executable]</tt> per node for a total of 8 processes, so mpirun will try to run mpi processes on 4 nodes
(OMP_NUM_THREADS is then probably best set at 4).
Your job script should still ask for these 4 nodes with the line
<source lang="bash">
#PBS -l nodes=4:ppn=8,walltime=....
</source>
(<tt>ppn=8</tt> is not a mistake here; the ppn parameter has a different meaning for PBS and for mpirun)

''The ppn parameter to ''mpirun'' is very important! Without it, eight mpi jobs would get bunched on the first node in this example, leaving 3 nodes unused.''

NOTE: In order to pin OpenMP threads inside the domain, use the corresponding OpenMP feature by setting the KMP_AFFINITY environment variable, see [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/optaps/common/optaps_openmp_thread_affinity.htm#KMP_AFFINITY_Environment_Variable|Intel's Compiler User and Reference Guide].

The IntelMPI manual is referenced on the front page of our wiki:

http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf

For the above example of a total of 8 processes on 4 nodes, you could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (hybrid job)
#
#PBS -l nodes=4:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2 intelmpi/4.1.2.040

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4

# PIN THE MPI DOMAINS ACCORDING TO OMP
export I_MPI_PIN_DOMAIN=omp

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -ppn 2 -np 8 ./a.out
</source>

'''Using Open MPI'''

For mixed MPI/OpenMP jobs using OpenMPI, which is the default for many users, the procedure is similar, but details differ.

* Request the number of nodes in the PBS script.
* Set OMP_NUM_THREADS to the number of threads per MPI process.
* In addition to the -np parameter for mpirun, add the argument <tt>--bynode</tt>, so that the mpi processes are not bunched up.

So for example, to start a total of 8 processes on 4 nodes, you could use the following script
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (hybrid job)
#
#PBS -l nodes=4:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel/15.0.2 openmpi/intel/1.6.4

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4

# EXECUTION COMMAND; -np = nodes*processes_per_nodes; --byhost forces a round robin of nodes.
mpirun -np 8 --bynode ./a.out
</source>

* More questions about running your jobs? See the [[FAQ#Running_your_jobs|FAQs Running your jobs]] and [[FAQ#Errors_in_running_jobs|Errors in running jobs]] sections.

===Automated Email Notifications from Jobs===

By default, you get an email if your job fails ('''a'''borts) when run through the scheduler. You can setup email notifications as well for when the jobs '''b'''egins or '''e'''nds. For this, you use the <tt>-m</tt> submission option, followed by any combination of '''a''' (to get emails when the job is aborted, e.g. by canceljob or qdel), '''b''' (to get emails when the job starts), and '''e''' (to get emails when the job ends), or '''n''' (for no emails except errors such as going over the walltime limit).

For instance, to get email in all three cases, you could have a job script like this:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC
#
#PBS -l nodes=2:ppn=8,walltime=1:00:00
#PBS -N test
#PBS -m abe

# load modules (must match modules used for compilation)
module load intel openmpi

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 16 ./a.out
</source>

Notes:

* Getting email for every job is not necessarily a good idea, as you may get a lot of email. This is why the default is to send an email only when the job fails

* The email address that the emails are sent to is the one that you gave when you created your account.


* Emails notifying that a job exceeded its walltime cannot be suppressed. The only way not to get them is to have your jobs stop before the walltime. If your job uses [[Checkpoints]] properly so that it can restart, it may be safe to have that job script self-terminate just before the walltime is over, by adding, at the top of the script, a line like this:

<source lang="bash">
sleep $((PBS_WALLTIME-60)) && (echo "out of time, self-terminating"; kill $$) &
</source>
which terminates the jobs script one minute before the wall time is over. You may want to combine this technique with [[Using Signals]].

===Submitting an Interactive (Debug) Job===

Your development work flow may require a lot of small test runs. You are allowed to do these on the development nodes, as long as it's very brief (a few minutes), and does not use all cores on the machine. For anything more you will have to use the compute nodes. (In a hurry? Check out the debugjob command below!)

It is sometimes convenient to run a job interactively; this can be very handy for debugging purposes. In this case, you type a <tt>qsub</tt> command which submits an interactive job to the queue; when the scheduler selects this job to run, then it starts a shell running on the first node of the job, which connects to your terminal. You can then type any series of commands (for instance, the same commands listed as in the batch submission script above) to run a job interactively.

For example, to start the same sort of job as in the batch submission script above, but interactively, one would type
<pre>
$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00
</pre>
If your interactive job requires graphics, add "-X" to this command.

Note that this is exactly the <tt>#PBS -l</tt> line in the batch script above (which requests all 8 processors on each of 2 nodes for one hour), but prepended with a <tt>-I</tt> for `interactive'. When this job begins, your terminal will now show you as being logged in to one of the compute nodes, and one can type in any shell command, run <tt>mpirun</tt>, etc. When you exit the shell, the job will end.

Interactive jobs can be used with any of the [[ Moab#GPC | GPC queues ]] however, there is a short
high turnover queue called [[ Moab#debug | debug ]] which can be especially useful when the system is busy:
<pre>
$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00 -q debug
</pre>

Because the combination of running interactively and use the high turnover queue is very common, SciNet has the specialized command
<pre>
$ debugjob [NUMBEROFNODES]
</pre>
where <tt>NUMBEROFNODES</tt> is an optional argument that defaults to one. This command requests NUMBEROFNODES compute nodes for a limited time between 30 minutes and 2 hours (less time the more nodes you take), and gives you a shell on the (first) compute node.

*More questions about running test jobs? See the [[FAQ#Testing_your_Code|FAQ]].



===QDR vs. DDR Infiniband===

The GPC Infiniband network GPC has two sections, one connecting 3,024 nodes with 5:1 blocking/oversubscribed QDR and the second connecting 840 nodes 1:1 non-blocking DDR Infiniband. By default a user's job will go to whichever network section best accommodates it, typically smaller jobs to the QDR and larger jobs to the DDR. However a user can override this by simply adding the flags "ddr" or "qdr" to the job resource request.

For example, to request two nodes anywhere on the GPC (QDR or DDR), use
<pre>
#PBS -l nodes=2:ppn=8,walltime=1:00:00
</pre>
in your job submission script.

For two nodes using DDR, use
<pre>
#PBS -l nodes=2:ddr:ppn=8,walltime=1:00:00
</pre>

To get two nodes using QDR, instead, you would use
<pre>
#PBS -l nodes=2:qdr:ppn=8,walltime=1:00:00
</pre>

The queueing system also tries its best to keep jobs within the same switch of the QDR thus avoiding the 5:1 blocking. A user can can explicitly request this behaviour if their jobs is less than 30 nodes using

<pre>
#PBS -W x=nodesetisoptional:false
</pre>

===HyperThreading===

Each GPC compute node has 8 Nehalem cores (2 sockets each with a four-core Intel Xeon E5540 @ 2.53GHz). Thus, to make full use of the computing power of a GPC node, you must be running least 8 "tasks" -- MPI processes, or OpenMP threads.

Under most circumstances, running exactly 8 tasks is the most efficient way to use these nodes. However, sometimes software design (eg, having one thread for communication and one for computation) can usefully `oversubscribe' the number of physical cores, and running (say) twice as many tasks as cores can be a useful strategy. If your code is highly memory-bandwidth bound, having one task ready to run while another waits for memory access can make more effective use of the processor.

The Nehalem processors have hardware support for such two-way overloading of processors, through "HyperThreading"; there are an extra set of registers on each core to facilitate rapid switching between two tasks, making it look to the operating system that there are in fact 16 cores per node. Depending on the nature of your code, making use of these virtual extra cores may speed up or slow down your computation; you should run small test cases before running production jobs in this manner. In most cases, the speed difference will be under 10%. Some of our users have obtained an 8% speedup by running gromacs with 16 tasks instead of 8 on a single node (mpirun -np 16 ./gromacs/mdrun -npme 4 is 108% the speed of mpirun -np 8 ./gromacs/mdrun with -npme 2 or -1).

====HyperThreading with OpenMP====

To use hyperthreading with an OpenMP job, one just runs twice as many threads as one would have previously; eg, if you were running 8 threads before (<tt>export OMP_NUM_THREADS=8</tt>) you would run with 16 (<tt>export OMP_NUM_THREADS=16</tt>). Everything else remains the same, including the job submission script; one still uses <tt>ppn=8</tt> in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

====HyperThreading with MPI====

To use hyperthreading with an MPI job, one just runs twice as many MPI processes as one would have previously; eg, if you were running on three nodes using 8 MPI tasks per node and used <tt>mpirun ... -np 24</tt>, you could run instead with <tt>-np 48</tt>. Everything else remains the same, including the job submission script; one still uses <tt>ppn=8</tt> in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

Note that if you are using OpenMPI (as is the default), there is another consideration; OpenMPI assumes that there is no oversubscription and each task very aggressively makes full use of a core when it is waiting for a message (eg, the waits are "busywaits"). If you find a significant slowdown when running multiple MPI tasks per core with OpenMPI, you may want to try adding the additional option to mpirun: <tt>--mca mpi_yield_when_idle 1</tt>. This will increase the latency of individual messages, but free up the core to do additional work while waiting.

With IntelMPI, the problem should be less pronounced, but you can still improve things by using <tt>mpirun -genv I_MPI_SPIN_COUNT 1 ...</tt>

'''Examples of hyperthreading with MPI'''

Hyperthreading using gromacs: https://support.scinet.utoronto.ca/wiki/index.php/Gromacs#Hyperthreading_with_Gromacs

====HyperThreading with Hybrid MPI/OpenMP codes====

With a hybrid code, one has extra flexibility in how to assign the "extra" cores -- you could run extra MPI tasks or extra OpenMPI threads. As with all hybrid codes, the combination which results in the best performance depends very strongly on the nature of your code, and you should experiment with different combinations. In addition, with hybrid codes processor and memory affinity issues become very important; if you're unsure as to how to tune your application for best performance, please make an appointment with the SciNet technical analysts for more help.

===Memory Configuration===

{| class="wikitable"
!{{Hl2}}|Number of Nodes
!{{Hl2}}|Memory
!{{Hl2}}|Notes
|-
|3655
|16 GB
|14 GB is available for your application.
|-
|205
|32 GB
|use the flag m32g to request (see below)
|-
|72
|64 GB
|part of the contributed system [[Sandy]], but unused cycles are available.
|-
|4
|128 GB or 256 GB
|2 x 128 GB and 2x256 GB as part of [[Sandy]],
|-
|2
|128 GB
|2 x older harpertown nodes with 16 cores and 128 GB each.
|-
|2
|128 GB
|2 x new haswell nodes with 20 cores and 128 GB each.
|}

'''16G'''

There are 3655 nodes which have 16G of memory, and is the primary configuration in the GPC. These nodes will be used by default.
On these nodes, about 2 GB is taken by the operating system. So for mpi runs with 8 processes per node, this leaves about 1.75GB max per mpi process. Do not try to use more than the available memory: the node will crash and your job will either die or hang until the requested walltime has elapsed.

If you need more memory per process or per thread, you can either try to use the limit number of larger memory nodes listed below, or you can run with less mpi processes, or use a different decomposition, such that the job fits on a node.



'''32G'''

There are 205 nodes which have 32G of memory. To request these nodes use:

<pre>
$ qsub -l nodes=1:m32g:ppn=8,walltime=1:00:00
</pre>

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total of 32GB.

'''64G/128G/256G'''

There are 72 16-core Intel Sandybridge nodes which have 64G, 2 with 128G, and 2 with 256G of memory available as part of the contributed [[Sandy]] cluster. These nodes are requested through the <tt>sandy</tt> queue.

<pre>
$ qsub -l nodes=1:m64g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m256g:ppn=16,walltime=1:00:00 -q sandy
</pre>

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total.

'''128G'''

There are four other stand-alone large memory (128GB) nodes which are primarily to be used for data analysis of runs. Both are Intel machines running the same linux OS as the compute nodes however, they are of different Intel generations than the regular GPC compute nodes. Two nodes are 16 core older Harpertown nodes, so codes may have to be compiled separately for these machine, and some modules that work on the other GPC nodes, such as octave, will not work on these nodes. The two other are 20 core newer Haswell nodes that all of the GPC modules should work on.

These nodes can be accessed using a specific <tt>largemem</tt> queue

<pre>
$ qsub -l nodes=1,walltime=1:00:00 -q largemem -I
</pre>

To specifically request the older/newer nodes, use the number of processers (16/20)

<pre>
$ qsub -l nodes=1:ppn=16,walltime=1:00:00 -q largemem -I
$ qsub -l nodes=1:ppn=20,walltime=1:00:00 -q largemem -I
</pre>

Use the latter reqeust to run any code compiled specifically for the GPC nodes (such as the python modules).

'''Note:''' To estimate your time of access to these nodes, use

<pre>
$ showq -w class=largemem
</pre>

====How to run fewer MPI processes per node for memory bound applications====

If your compute job requires more memory than is available, it will crash. In some cases, this will even crash the node itself.
One way to exceed the memory is if your MPI application is too large for 8 processes to fit in memory. Oftentimes, you can solve this problem by subdividing the computation over more processes (reducing the memory per process) and ask for more nodes. Sometimes, this does not work, for instance if each MPI process requires a fixed minimum chunk of memory that is too large. In that case, you will want to run with less mpi processes than there are cpus in a nodes (i.e. less than 8 for the GPC). Running in such a mode requires a change in the mpirun command. The mpirun command now should not only specify the total number of mpi processes, but should also specify how these processes will be distributed over the nodes. The default distribution of processes is such that the first nodes get loaded with 8 processes, than the second nodes gets filled up, etc, and this will clearly lead to the same memory issue that you had before. Instead, you would want your processes to be distributed evenly over the nodes. The syntax for specifying that depends on whether you are using openmpi or intelmpi. For openmpi running on 4 nodes with 3 processes per node, for example, you would invoke mpirun as follows
<pre>
mpirun -np 12 --bynode <application>
</pre>
For intelmpi, the same setup would be invoked using:
<pre>
mpirun -np 12 -ppn 3 <application>
</pre>

====[https://wiki.scinet.utoronto.ca/wiki/index.php/Moab#Checking_the_memory_usage_from_jobs Checking memory usage from jobs]====

===Ram Disk===

On the GPC nodes, there is a `ram disk' available - up to half of the memory on the node may be used as a temporary file system. This is particularly useful for use in the early stages of migrating destop-computing codes to a High Performance Computing platform such as the GPC. It is much faster than real disk and does not require network traffic; however, each node sees its own ramdisk and cannot see files on that of other nodes. This is a very easy way to cache writes (by writing them to fast ram disk instead of slow `real' disk); and then one would periodically copy the files to files on /scratch or /project so that they are available after the job has completed.

To use the ramdisk, create and read to or write from files in /dev/shm/... just as one would to (eg) $SCRATCH. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

NOTE: it is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].

=== Managing jobs on the Queuing system ===

Information on checking available resources, starting, viewing, managing and canceling jobs on [[Moab | Moab/Torque]]. Also check out the [[FAQ#Monitoring_jobs_in_the_queue|FAQ]].

Oldwiki.scinet.utoronto.ca:System Alerts

2016-09-25T13:07:47Z

Nolta: /* System Status */

== System Status==

{|
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:down.png|up]]File System
|-
|[[File:down.png|up|link=Gravity]][[Gravity]]
|[[File:down.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:down.png|down|link=BGQ]][[BGQ]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|}

Sun Sep 25 08:46 EDT 2016 Power outage at datacentre.

Thu Sep 8 10:46:22 EDT 2016 Until further notice HPSS jobs are being manually throttled to ensure the cache does not get overwhelmed. We have new tape drives being added to the system soon, and that will help alleviate the problem when we return to normal operations.

Fri Sep 2 17:31:44 EDT 2016 The HPSS disk-based cache is full, and currently being drained to tape. Until enough material has been migrated and purged the system will remain on hold.

Sun Aug 28 16:10:56 EDT BGQ racks were shutoff due to environment issue, system being restarted. Please resubmit your jobs.

Sun 28 Aug 2016 10:19:17 EDT Turning systems on. Expect to be up in a couple of hours.

Sun Aug 28 09:16:02 EDT 2016 Datacentre was shutdown due to facility power failure.

Tue 16 Aug 2016 21:55:20 Cooling restored, filesystem up and OK, bringing up clusters. GPC should be available to users by 11PM (perhaps as early as 1030)

Tue 16 Aug 2016 20:46:37 Water service has been restored to building. Restarting cooling system.

Tue 16 Aug 2016 19:52:32 SciNet-related maintenance and modifications have been completed successfully. Work on the building water valve is expected to be done on time (by 9PM). Once water service is restored we need to restore cooling, power-up filesystems and then restart the clusters. Unlikely that any systems are available to users before 11PM and it will take longer to get everything online. Check here for updates

Tue 16 Aug 2016 07:07:32 Shutdown has started

 Scheduled full-day maintenance shutdown begins:

7AM, Tuesday, 16 Aug 

Several projects (adding new 208V circuits for storage, cooling tower maintenance etc) are being carried out on same day as the landlord needs to shutdown the main building water supply (and therefore our cooling system as well) for repairs.

Expect to start bringing systems up about 10PM depending on when the water work is done. Check here for further updates during the day

Oldwiki.scinet.utoronto.ca:System Alerts

2016-09-25T13:05:09Z

Nolta: /* System Status */

== System Status==

{|
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:down.png|up]]File System
|-
|[[File:down.png|up|link=Gravity]][[Gravity]]
|[[File:down.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:down.png|down|link=BGQ]][[BGQ]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|}

Sun Sep 25 09:04 EDT 2016 Power outage at datacentre.

Thu Sep 8 10:46:22 EDT 2016 Until further notice HPSS jobs are being manually throttled to ensure the cache does not get overwhelmed. We have new tape drives being added to the system soon, and that will help alleviate the problem when we return to normal operations.

Fri Sep 2 17:31:44 EDT 2016 The HPSS disk-based cache is full, and currently being drained to tape. Until enough material has been migrated and purged the system will remain on hold.

Sun Aug 28 16:10:56 EDT BGQ racks were shutoff due to environment issue, system being restarted. Please resubmit your jobs.

Sun 28 Aug 2016 10:19:17 EDT Turning systems on. Expect to be up in a couple of hours.

Sun Aug 28 09:16:02 EDT 2016 Datacentre was shutdown due to facility power failure.

Tue 16 Aug 2016 21:55:20 Cooling restored, filesystem up and OK, bringing up clusters. GPC should be available to users by 11PM (perhaps as early as 1030)

Tue 16 Aug 2016 20:46:37 Water service has been restored to building. Restarting cooling system.

Tue 16 Aug 2016 19:52:32 SciNet-related maintenance and modifications have been completed successfully. Work on the building water valve is expected to be done on time (by 9PM). Once water service is restored we need to restore cooling, power-up filesystems and then restart the clusters. Unlikely that any systems are available to users before 11PM and it will take longer to get everything online. Check here for updates

Tue 16 Aug 2016 07:07:32 Shutdown has started

 Scheduled full-day maintenance shutdown begins:

7AM, Tuesday, 16 Aug 

Several projects (adding new 208V circuits for storage, cooling tower maintenance etc) are being carried out on same day as the landlord needs to shutdown the main building water supply (and therefore our cooling system as well) for repairs.

Expect to start bringing systems up about 10PM depending on when the water work is done. Check here for further updates during the day

Oldwiki.scinet.utoronto.ca:System Alerts

2016-07-29T20:45:20Z

Nolta: /* System Status */

File:Intro to the unix shell.pdf

2016-07-11T17:07:17Z

Nolta:

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T17:06:51Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [https://support.scinet.utoronto.ca/~nolta/HPC_intro.pdf Slides]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

* [[Media:intro_to_the_unix_shell.pdf | Slides ]]

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T17:05:07Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [https://support.scinet.utoronto.ca/~nolta/HPC_intro.pdf Slides]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T17:04:48Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [https://support.scinet.utoronto.ca/~nolta/HPC_intro.pdf | Slides]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T17:04:12Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [[https://support.scinet.utoronto.ca/~nolta/HPC_intro.pdf | Slides]]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T17:03:58Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [[:https://support.scinet.utoronto.ca/~nolta/HPC_intro.pdf | Slides]]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T17:03:28Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [[Media:https://support.scinet.utoronto.ca/~nolta/HPC_intro.pdf | Slides]]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

2016 Ontario Summer School for High Performance Computing Central

2016-07-11T16:20:49Z

Nolta: /* Slides and other material */

===About===

The 2016 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 11th to July 15th, 2016, in Toronto.

The location of this version of the summer school is:

'''Wallberg Building''' 
'''Rooms 116 and 119''' 
University of Toronto 
St. George Campus 
184-200 College Street 
Toronto, Ontario, M5S 3E5 
Canada

More information can be found below and in the [[Media:handout2016.pdf|handout package]].

__TOC__

===Introduction to HPC and SciNet===

=====Abstract=====
This half-day session will provide an introduction to basic concepts of high-performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 11, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic Linux skills are useful to follow along. If you do not have these, consider taking the "Intro to the Linux Shell" in the afternoon.

=====Slides and other material=====

* [[Media:HPC_intro_2016.pdf | Slides]]

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In these two half-day sessions, lectures and hands-on labs are interspersed, and the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Monday July 11, 2016, 13:30 - 16:30 
Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====

* [[Media:OpenMP2016.pdf | Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Introduction to the Linux Shell===

====Abstract====
Working with many of the HPC systems in Ontario involves using the Linux/UNIX command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. In this half-day session, you can become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 11, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====

----

===R for Data Science===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Tuesday July 12, 2016, 09:30 - 12:30

''Location''

Wallberg Building, Room 119 

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with R or Rstudio installed.

=====Slides and other material=====

----

===Programming Clusters with MPI===

=====Abstract=====
In this 1.5-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30 
Wednesday July 13, 2016, 09:30 - 12:30 
Wednesday July 13, 2016, 13:30 - 16:30 

''Location''

Wallberg Building, Room 116

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
C and/or Fortran scientific programming; experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[File:MPIssc2016.pdf | Slides]]
----

===Parallel R for Data Science===

=====Abstract=====
This half-day session will cover parallel programming in R, with a focus on parallel data analysis. Topics covered will include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Tuesday July 12, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====

----

===Python for Scientific Computing===

=====Abstract=====
In this half-day session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Wednesday July 13, 9:30 - 12:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Python for High Performance Computing===

=====Abstract=====
This half-day session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Wednesday July 13, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====

----

===Programming GPUs with CUDA===

=====Abstract=====
This two-day session is an introductory course covering programming and computing on GPUs - graphics processing units - which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30 
Friday July 15, 2016, 09:30 - 12:30 
Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 116

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====

----

===Visualization===

=====Abstract=====
The first part of this whole-day session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The afternoon part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Thursday July 14, 2016, 09:30 - 12:30 
Thursday July 14, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, xmgr ([http://plasma-gate.weizmann.ac.il/Grace/ Grace],[http://plasma-gate.weizmann.ac.il/Xmgr/ xmgr]), ssh (graphic client), your favorite vnc client (egs. [https://www.realvnc.com/download/viewer/ RealVNC],[http://www.tightvnc.com/download.php TightVNC]), python and matplotlib ([http://matplotlib.org/ matplotlib]); afternoon session: ParaView ([http://www.paraview.org/download/ ParaView]) and VisIt ([https://wci.llnl.gov/simulation/computer-codes/visit/executables VisIt]).

=====Slides and other material=====
======Slides======
======Datasets======
[ Morning datasets and scripts]

[https://support.scinet.utoronto.ca/~mponce/ss2016/datasets.tar.gz Afternoon dataset]

[https://support.scinet.utoronto.ca/~mponce/ss2016/aneurysm_data.tar.gz Aneurysm dataset]
----

===Debugging===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. In this half-day session, we will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems.

=====Details=====
''Date/time''

Friday July 15, 09:30 - 12:30

''Location''

Wallberg Building, Room 119

Note the different room number!

''Instructors''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====

----

===Bring Your Own Code===

=====Abstract=====
The final afternoon of the summer school is your chance to get help with your own code from the SciNet software analysts. Bring your code, explain your problem or concerns, and we'll give you advice and work with you to try to improve it.

=====Details=====
''Date/time''

Friday July 15, 2016, 13:30 - 16:30

''Location''

Wallberg Building, Room 119

''Instructors''

Various SciNet software analysts.

=====Prerequisites=====
None.

Python

2016-05-19T22:09:41Z

Nolta: /* Python on the GPC */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have several versions of python installed, compiled against fast intel math libraries. To load the python modules, type the following commands:

{|
! Version
! Command
|-
|2.7.2
|<tt>module load gcc intel python</tt>
|-
|2.7.3
|<tt>module load gcc intel/13.1.1 python/2.7.3</tt>
|-
|2.7.5
|<tt>module load gcc intel/13.1.1 python/2.7.5</tt>
|-
|2.7.8
|<tt>module load intel/15.0.2 python/2.7.8</tt>
|-
|2.7.11
|<tt>module load anaconda2/4.0.0</tt>
|-
|3.3.4
|<tt>module load gcc intel/14.0.1 python/3.3.4</tt>
|-
|3.5.1
|<tt>module load anaconda3/4.0.0</tt>
|}

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/2.7.8
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.9.1
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support. No longer supported.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 2.3.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.4.2
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded.
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| -
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded.
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module. No longer supported.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.1
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| -
| -
| -
| -
| Python interface to the svn version control system.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| 3.2
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.21.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.4
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 7.0
| 5.1
| Enables easy installation of new python modules
|-
| [http://pandas.pydata.org/ pandas]
| 0.13.0
| 0.13.0
| 0.13.0
| 0.15.0
| 0.14.1
| high-performance, easy-to-use data structures and data analysis tools.
|-
| [http://www.astropy.org astropy]
| -
| -
| 0.3
| 0.4.2
| 0.3.2
| astronomical routines
|-
| [http://briansimulator.org/ brian]
| 1.4.1
| 1.4.1
| 1.4.1
| 1.4.1
| -
| spiking neural network simulator
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2016-05-19T22:05:10Z

Nolta: add python v 2.7.8, 2.7.11, 3.5.1

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2, 2.7.3, 2.7.5, and 3.3.4 installed, compiled against fast intel math libraries. To load the python modules, type the following commands:

{|
! Version
! Command
|-
|2.7.2
|<tt>module load gcc intel python</tt>
|-
|2.7.3
|<tt>module load gcc intel/13.1.1 python/2.7.3</tt>
|-
|2.7.5
|<tt>module load gcc intel/13.1.1 python/2.7.5</tt>
|-
|2.7.8
|<tt>module load intel/15.0.2 python/2.7.8</tt>
|-
|2.7.11
|<tt>module load anaconda2/4.0.0</tt>
|-
|3.3.4
|<tt>module load gcc intel/14.0.1 python/3.3.4</tt>
|-
|3.5.1
|<tt>module load anaconda3/4.0.0</tt>
|}

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/2.7.8
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.9.1
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support. No longer supported.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 2.3.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.4.2
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded.
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| -
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded.
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module. No longer supported.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.1
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| -
| -
| -
| -
| Python interface to the svn version control system.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| 3.2
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.21.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.4
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 7.0
| 5.1
| Enables easy installation of new python modules
|-
| [http://pandas.pydata.org/ pandas]
| 0.13.0
| 0.13.0
| 0.13.0
| 0.15.0
| 0.14.1
| high-performance, easy-to-use data structures and data analysis tools.
|-
| [http://www.astropy.org astropy]
| -
| -
| 0.3
| 0.4.2
| 0.3.2
| astronomical routines
|-
| [http://briansimulator.org/ brian]
| 1.4.1
| 1.4.1
| 1.4.1
| 1.4.1
| -
| spiking neural network simulator
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

SciNet User Support Library

2016-05-11T15:04:40Z

Nolta: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* May 11: GPC: https://my.scinet.utoronto.ca/ is available for testing.

* May 3: GPC: Versions 15.0.6 and 16.0.3 of the Intel Compilers are installed as modules.

* Feb 12: GPC: Version 6.0 of Allinea Forge (DDT Debugger, MAP, Performance Reports) installed as a module.

* Jan 11: The 2016 Resource Allocations for compute cycles are now in effect.

* Nov 23: The quota for home directories has been increased from 10 GB to 50 GB.

* Nov 23, GPC: Two [[Visualization Nodes]], viz01 and viz02, are being setup. They are 8-core Nehalem nodes with 2 graphics cards each, 64 GB of memory, and about 60GB of local hard disk. For now, you can directly log into viz01 to try it out. We would value users' feedback and request for suitable software, help with visualization projects etc.

* Nov 16: ARC being decommisioned. During a transition period, the ARC head node and two compute nodes will be kept up. Users are encouraged to start using [[Gravity]] instead.

* Nov 12, GPC: The number of [[GPC_Quickstart#Compile.2FDevel_Nodes | GPC devel]] nodes has been doubled from 4 to 8, and the new ones can be accessed using gpc0[5-8].

* Nov 12, GPC: Two new [[GPC_Quickstart#Memory_Configuration| 128 GB RAM Haswell]] nodes have been added to the "largemem" queue.

* Sept 7, GPC: The number of nodes with [[GPC_Quickstart#Memory_Configuration|32 GB of RAM]] has been increased from 84 to 205.

* July 24, GPC: GCC 5.2.0 with [[Co-array_Fortran_on_the_GPC|Coarray Fortran]] support, installed as a module.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[TCS_Quickstart|TCS: Tightly Coupled System]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [[File:Rss.gif|link=http://support.scinet.utoronto.ca/podcasts.xml]] [http://support.scinet.utoronto.ca/podcasts.xml SciNet Training Webcasts]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

Oldwiki.scinet.utoronto.ca:System Alerts

2016-01-14T21:33:01Z

Nolta: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2015-12-17T14:51:16Z

Nolta: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2015-12-16T23:10:20Z

Nolta: file system down

2015 Ontario Summer School for High Performance Computing Central

2015-07-17T13:52:50Z

Nolta: /* Slides and other material */

[[File:Summerschool_hpc_2015.gif]]

===About===

The 2015 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 13th to July 17th, 2015, in Toronto.

===Group Photo===

[[File:GroupPhotoSS2015.jpg|800px]]



__TOC__

===Introduction to the Linux Shell===

====Abstract====

Working with many of the HPC systems in Ontario involves using the linux/unix command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. Become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====
* [[Media:SS2015_Introduction_to_the_Unix_Shell.pdf|Slides]]

----

===Introduction to High Performance Computing===

=====Abstract=====
This session will provide an introduction to basic concepts of high performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic Linux shell skills Registration for all summer school sessions will become available at https://www.sharcnet.ca/events/ss2015.

=====Slides and other material=====
[http://www.scinethpc.ca/~mponce/ss2015/HPC_intro-2015.pdf Slides]

----

===Data Analysis with R===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with a R and Rstudio installed.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/7/73/Intro-R.pdf Slides].

----

===Parallel R===

=====Abstract=====
This session will cover parallel programming R, with a focus on parallel data analysis. Topics covered include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/a/ac/Parallel-R.pdf Slides].

----

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In this session, in which lectures and hands-on labs are interspersed, the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Tuesday July 14, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====
* [[Media:OpenMP_SS2015.pdf|Slides]]
* [[Media:OpenMP_SS2015_code.tar.gz|Code]]

----

===Scientific Computing with Python===

=====Abstract=====
In this session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Tuesday July 14, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/f/f3/Intro-Python.pdf Slides].

----

===Python for High Performance Computing===

=====Abstract=====
This session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Tuesday July 14, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Python programming experience and a rudimentary understanding of parallel computing.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/2/2b/Parallel-Python.pdf Slides].

----

===Programming Clusters with MPI===

=====Abstract=====
In this two-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
Programming experience with Fortran or C/C++.

=====Slides and other material=====
The slides for this class can be found [http://support.scinet.utoronto.ca/~northrup/SS2015 here].

----

===Programming GPUs with CUDA===

=====Abstract=====
This is an introductory course covering programming and computing on GPUs---graphics processing units---which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA-C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====
''Day 1:''
* [[Media:CUDA_summer_school_2015_central.pdf|Slides]]
* [[Media:CUDA_day1.tar.gz|Code]]
See also [http://ppomorsk.sharcnet.ca http://ppomorsk.sharcnet.ca]

''Day 2:''
* [[Media:CUDA2.pdf|Slides]]

----

===Visualization===

=====Abstract=====
The first part of this session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The second part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, ssh (graphic client), vnc client, python and matplotlib; afternoon session: ParaView & VisIt.

=====Slides and other material=====
Morning session: [http://www.scinethpc.ca/~mponce/ss2015/ss2015_visualization-morning.pdf Visualization I]

Afternoon session: [http://www.scinethpc.ca/~mponce/ss2015/ss2015_visualization-afternoon.pdf Visualization II]

Samples to use during the tutorial: [http://www.scinethpc.ca/~mponce/ss2015/data.tar.gz Data sets]

To come: An example of a complete visualization session in VisIt
----

===Debugging, Optimization, and Best Practices===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. We will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems. Next, we will present general issues, common pitfalls and optimization strategies that are application to HPC systems. By using profiling tools available for parallel programs, we can see when the bottlenecks are. We can try to alleviate them by following common best practices, allowing one to obtain their results more quickly and at the same time minimize their burden on shared system resources. Walk-through examples and case studies will be presented during the session to illustrate the concepts.

=====Details=====
''Date/time''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''

Dr. Ramses van Zon and Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====
* [[Media:DebugSS2015.pdf | Slides for the debugging session]]
* [[Media:DebugSS2015code.tar.gz | Debuggable code ]]
* [[Media:SSHPC_OBP2013.pdf|Slides for the best practices session]]

2015 Ontario Summer School for High Performance Computing Central

2015-07-14T13:47:50Z

Nolta:

[[File:Summerschool_hpc_2015.gif]]

===About===

The 2015 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 13th to July 17th, 2015, in Toronto.



__TOC__

===Introduction to the Linux Shell===

====Abstract====

Working with many of the HPC systems in Ontario involves using the linux/unix command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. Become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====
* [[Media:SS2015_Introduction_to_the_Unix_Shell.pdf|Slides]]

===Introduction to High Performance Computing===

=====Abstract=====
This session will provide an introduction to basic concepts of high performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic Linux shell skills Registration for all summer school sessions will become available at https://www.sharcnet.ca/events/ss2015.

=====Slides and other material=====
[http://www.scinethpc.ca/~mponce/HPC_intro-2015.pdf Slides]

===Data Analysis with R===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with a R and Rstudio installed.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/7/73/Intro-R.pdf Slides].

===Parallel R===

=====Abstract=====
This session will cover parallel programming R, with a focus on parallel data analysis. Topics covered include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/a/ac/Parallel-R.pdf Slides].

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In this session, in which lectures and hands-on labs are interspersed, the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Tuesday July 14, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====
[[Media:OpenMP_SS2015.pdf|Slides]]

===Scientific Computing with Python===

=====Abstract=====
In this session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Tuesday July 14, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/f/f3/Intro-Python.pdf Slides].

===Python for High Performance Computing===

=====Abstract=====
This session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Tuesday July 14, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Python programming experience and a rudimentary understanding of parallel computing.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/2/2b/Parallel-Python.pdf Slides].

===Programming Clusters with MPI===

=====Abstract=====
In this two-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
Programming experience with Fortran or C/C++.

=====Slides and other material=====
To come

===Programming GPUs with CUDA===

=====Abstract=====
This is an introductory course covering programming and computing on GPUs---graphics processing units---which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA-C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====
To come

===Visualization===

=====Abstract=====
The first part of this session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The second part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, ssh (graphic client), vnc client, python and matplotlib; afternoon session: paraview.

=====Slides and other material=====
To come

===Debugging, Optimization, and Best Practices===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. We will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems. Next, we will present general issues, common pitfalls and optimization strategies that are application to HPC systems. By using profiling tools available for parallel programs, we can see when the bottlenecks are. We can try to alleviate them by following common best practices, allowing one to obtain their results more quickly and at the same time minimize their burden on shared system resources. Walk-through examples and case studies will be presented during the session to illustrate the concepts.

=====Details=====
''Date/time''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''

Dr. Ramses van Zon and Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====
To come

File:SS2015 Introduction to the Unix Shell.pdf

2015-07-13T16:54:09Z

Nolta:

2015 Ontario Summer School for High Performance Computing Central

2015-07-13T16:53:49Z

Nolta:

[[File:Summerschool_hpc_2015.gif]]

===About===

The 2015 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 13th to July 17th, 2015, in Toronto.



__TOC__

===Introduction to the Linux Shell===

====Abstract====

Working with many of the HPC systems in Ontario involves using the linux/unix command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. Become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====
* [[Media:SS2015_Introduction_to_the_Unix_Shell.pdf|Slides]]
* [[Media:SS2014_IntroToShell.tgz|Files]]

===Introduction to High Performance Computing===

=====Abstract=====
This session will provide an introduction to basic concepts of high performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic Linux shell skills Registration for all summer school sessions will become available at https://www.sharcnet.ca/events/ss2015.

=====Slides and other material=====
[http://www.scinethpc.ca/~mponce/HPC_intro-2015.pdf Slides]

===Data Analysis with R===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with a R and Rstudio installed.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/7/73/Intro-R.pdf Slides].

===Parallel R===

=====Abstract=====
This session will cover parallel programming R, with a focus on parallel data analysis. Topics covered include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====
[https://wiki.scinet.utoronto.ca/wiki/images/a/ac/Parallel-R.pdf Slides].

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In this session, in which lectures and hands-on labs are interspersed, the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Tuesday July 14, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====
To come

===Scientific Computing with Python===

=====Abstract=====
In this session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Tuesday July 14, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====
To come

===Python for High Performance Computing===

=====Abstract=====
This session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Tuesday July 14, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Python programming experience and a rudimentary understanding of parallel computing.

=====Slides and other material=====
To come

===Programming Clusters with MPI===

=====Abstract=====
In this two-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
Programming experience with Fortran or C/C++.

=====Slides and other material=====
To come

===Programming GPUs with CUDA===

=====Abstract=====
This is an introductory course covering programming and computing on GPUs---graphics processing units---which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA-C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====
To come

===Visualization===

=====Abstract=====
The first part of this session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The second part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, ssh (graphic client), vnc client, python and matplotlib; afternoon session: paraview.

=====Slides and other material=====
To come

===Debugging, Optimization, and Best Practices===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. We will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems. Next, we will present general issues, common pitfalls and optimization strategies that are application to HPC systems. By using profiling tools available for parallel programs, we can see when the bottlenecks are. We can try to alleviate them by following common best practices, allowing one to obtain their results more quickly and at the same time minimize their burden on shared system resources. Walk-through examples and case studies will be presented during the session to illustrate the concepts.

=====Details=====
''Date/time''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''

Dr. Ramses van Zon and Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====
To come

2015 Ontario Summer School for High Performance Computing Central

2015-07-13T07:01:49Z

Nolta: /* Slides and other material */

[[File:Summerschool_hpc_2015.gif]]

===About===

The 2015 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 13th to July 17th, 2015, in Toronto.



__TOC__

===Introduction to the Linux Shell===

====Abstract====

Working with many of the HPC systems in Ontario involves using the linux/unix command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. Become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====
* To come
* [[Media:SS2014_IntroToShell.tgz|Files]]

===Introduction to High Performance Computing===

=====Abstract=====
This session will provide an introduction to basic concepts of high performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic Linux shell skills Registration for all summer school sessions will become available at https://www.sharcnet.ca/events/ss2015.

=====Slides and other material=====
To come

===Data Analysis with R===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with a R and Rstudio installed.

=====Slides and other material=====
To come

===Parallel R===

=====Abstract=====
This session will cover parallel programming R, with a focus on parallel data analysis. Topics covered include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====
To come

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In this session, in which lectures and hands-on labs are interspersed, the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Tuesday July 14, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====
To come

===Scientific Computing with Python===

=====Abstract=====
In this session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Tuesday July 14, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====
To come

===Python for High Performance Computing===

=====Abstract=====
This session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Tuesday July 14, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Python programming experience and a rudimentary understanding of parallel computing.

=====Slides and other material=====
To come

===Programming Clusters with MPI===

=====Abstract=====
In this two-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
Programming experience with Fortran or C/C++.

=====Slides and other material=====
To come

===Programming GPUs with CUDA===

=====Abstract=====
This is an introductory course covering programming and computing on GPUs---graphics processing units---which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA-C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====
To come

===Visualization===

=====Abstract=====
The first part of this session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The second part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, ssh (graphic client), vnc client, python and matplotlib; afternoon session: paraview.

=====Slides and other material=====
To come

===Debugging, Optimization, and Best Practices===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. We will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems. Next, we will present general issues, common pitfalls and optimization strategies that are application to HPC systems. By using profiling tools available for parallel programs, we can see when the bottlenecks are. We can try to alleviate them by following common best practices, allowing one to obtain their results more quickly and at the same time minimize their burden on shared system resources. Walk-through examples and case studies will be presented during the session to illustrate the concepts.

=====Details=====
''Date/time''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''

Dr. Ramses van Zon and Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====
To come

2015 Ontario Summer School for High Performance Computing Central

2015-07-13T06:58:24Z

Nolta:

[[File:Summerschool_hpc_2015.gif]]

===About===

The 2015 Ontario Summer School on High Performance Computing, whose second installment of the year will be hosted by SciNet, provides attendees with opportunities to learn and share knowledge and experience in high performance and technical computing. The format: a five-day workshop with mixed lectures and hands-on sessions on a number of selected subjects, including shared memory programming, distributed memory programming, parallel data processing, and general purpose graphics processing unit programming.

Below are descriptions and course material for the Central version, given from July 13th to July 17th, 2015, in Toronto.



__TOC__

===Introduction to the Linux Shell===

====Abstract====

Working with many of the HPC systems in Ontario involves using the linux/unix command line. This provides a very powerful interface, but it can be quite daunting for the uninitiated. Become initiated with this course. This hands on session will cover basic commands and scripting, as well as touching on some powerful constructs like regular expressions. It could be a great boon for your productivity!

====Details====
''Time/date''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Mike Nolta (SciNet)

====Prerequisites====
A laptop with a linux-like environment.

====Slides and other material====
To come
[[File:SS2014_IntroToShell.tar.gz]]

===Introduction to High Performance Computing===

=====Abstract=====
This session will provide an introduction to basic concepts of high performance computing. It is intended to be a high level primer for those largely new to HPC, and serve as a foundation upon which to build over the coming days. Topics will include motivation for HPC, essential issues, problem characteristics as they apply to parallelism and a high level overview of parallel programming models. Strategies of running large sets of serial processes using e.g. GNU parallel, will also be presented.

=====Details=====
''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Basic Linux shell skills Registration for all summer school sessions will become available at https://www.sharcnet.ca/events/ss2015.

=====Slides and other material=====
To come

===Data Analysis with R===

=====Abstract=====
This session offers a brief introduction to R, with a focus on data analysis and statistics.

=====Details=====

''Date/time''

Monday July 13, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====

Some programming experience in any language. You should have used and written functions. Bring a laptop with a R and Rstudio installed.

=====Slides and other material=====
To come

===Parallel R===

=====Abstract=====
This session will cover parallel programming R, with a focus on parallel data analysis. Topics covered include snow, parallel, and foreach/doparallel.

=====Details=====

''Time/date''

Monday July 13, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Experience with programming in R and a rudimentary understanding of parallel computing.

=====Slides and other material=====
To come

===Programming Shared Memory Systems with OpenMP===

=====Abstract=====
In this session, in which lectures and hands-on labs are interspersed, the students will learn the basics of shared memory programming with OpenMP. In particular, we will discuss the OpenMP's threads, memory, and performance, reductions and load balancing. We will also discuss extensions to heterogeneous systems such as offered by the OpenMP 4.0 and OpenACC standards.

=====Details=====

''Time/date''

Tuesday July 14, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Ramses van Zon (SciNet)

=====Prerequisites=====
C/C++ and/or Fortran scientific programming; Experience editing and compiling code in a Linux environment.

=====Slides and other material=====
To come

===Scientific Computing with Python===

=====Abstract=====
In this session, you will be taught how to use python for research computing purposes. Topics include: the basics of python, automation, numpy, scipy, file i/o, and visualization.

=====Details=====
''Time/date'''

Tuesday July 14, 9:30 - 12:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
A bit of coding experience in any programming or scripting language. Command-line experience in Linux is a plus.

=====Slides and other material=====
To come

===Python for High Performance Computing===

=====Abstract=====
This session will cover parallel programming in python, with a focus on parallel data analysis. We will cover subprocess, multiprocessing, pypar and other parallel-enabling python packages.

=====Details=====
''Time/date''

Tuesday July 14, 13:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Erik Spence (SciNet)

=====Prerequisites=====
Python programming experience and a rudimentary understanding of parallel computing.

=====Slides and other material=====
To come

===Programming Clusters with MPI===

=====Abstract=====
In this two-day session, through lectures interspersed with hands-on labs, the students will learn the basics of MPI programming. Examples and exercises will be based on parallelization of common scientific computing problems.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''

Dr. Scott Northrup (SciNet)

=====Prerequisites=====
Programming experience with Fortran or C/C++.

=====Slides and other material=====
To come

===Programming GPUs with CUDA===

=====Abstract=====
This is an introductory course covering programming and computing on GPUs---graphics processing units---which are an increasingly common presence in massively parallel computing architectures. This session will cover both of the available C-like programming frameworks: NVIDIA’s CUDA-C. The basics of GPU programming will be covered, and students will work through a number of hands on examples. Demonstrations of profiling and debugging applications running on the GPU will also be included. The structuring of data and computations that makes full use of the GPU will be discussed in detail. Students should be able to leave the course with the knowledge necessary to begin developing their own GPU applications.

=====Details=====
''Time/date''

Wednesday July 15, 9:30 - 16:30

Thursday July 16, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''
Dr. Pawel Pomorski and Dr. Sergey Mashchenko (SHARCNET)

=====Prerequisites=====
C/C++ scientific programming, experience editing and compiling code in a Linux environment. Some experience with CUDA and/or OpenMP a plus.

=====Slides and other materials=====
To come

===Visualization===

=====Abstract=====
The first part of this session will be a basic introduction to general-purpose scientific visualization tools such as gnuplot, grace(xmgr), remote tools (tunneling, VNC), python's matplotlib, and examples from other. The second part will be a mix of lecture and hands-on to introduce more specialized scientific visualization software such as vmd, paraview, and visit.

=====Details=====
''Time/date''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 404

35 St. George Street

Toronto, ON M5S 1A4

''Instructor''
Dr. Marcelo Ponce (SciNet)

=====Prerequisites=====
Software installed, for the morning session: gnuplot, ssh (graphic client), vnc client, python and matplotlib; afternoon session: paraview.

=====Slides and other material=====
To come

===Debugging, Optimization, and Best Practices===

=====Abstract=====
Debugging is an important step in developing a new code, or porting an old one to a new machine. We will discuss the debugging of frequently encountered bugs in serial code with gdb and the debugging of parallel (MPI and threaded) codes with DDT on live systems. Next, we will present general issues, common pitfalls and optimization strategies that are application to HPC systems. By using profiling tools available for parallel programs, we can see when the bottlenecks are. We can try to alleviate them by following common best practices, allowing one to obtain their results more quickly and at the same time minimize their burden on shared system resources. Walk-through examples and case studies will be presented during the session to illustrate the concepts.

=====Details=====
''Date/time''

Friday July 17, 9:30 - 16:30

''Location''

Galbraith Building, Room 405

35 St. George Street

Toronto, ON M5S 1A4

''Instructors''

Dr. Ramses van Zon and Dr. Mike Nolta (SciNet)

=====Prerequisites=====
Basic familiarity with a high-level language (C/C++/Fortran) as well as compiling and running programs via the command line environment on a HPC system.

=====Slides and other material=====
To come

File:SS2014 IntroToShell.tar.gz

2015-07-13T06:50:51Z

Nolta:

SciNet Command Line Utilities

2015-05-03T16:22:35Z

Nolta: remove scinetUsage

__TOC__

Below is list of handy command-line utilities written by SciNet to help manage your data and compute. Currently, on the GPC you need to have the '''extras''' module loaded for these to work. Note that in the table, optional arguments are denoted with square brackets (i.e. ''''[]''''), while mandatory arguments are denoted with angular brackets (i.e. ''''<>'''').

= Storage =
{| border="1" cellpadding="5" cellspacing="0"
!{{Hl2}} | Command
!{{Hl2}}| Arguments
!{{Hl2}}| Description
!{{Hl2}}| Cluster
|-
|quota
|
|Short overview of a user's storage usage.
|GPC
|-
|diskUsage
|''-h'' to see all command options
|Informs about the user and group file system usage.
|GPC
|-
|topUserDirOver1000list
|
|Lists your directories that have over 1,000 files
|GPC
|-
|topUserDirOver1GBlist
|
|Lists your directories that have over 1 GB of data
|GPC
|-
|}

= Jobs & Queues =
{| border="1" cellpadding="5" cellspacing="0"
!{{Hl2}} | Command
!{{Hl2}}| Arguments
!{{Hl2}}| Description
!{{Hl2}}| Cluster
|-
|qsum
|most showq arguments work
|Lists jobs running or in the queue, grouped by user.
|GPC,TCS
|-
|llq1
|
|Detailed information on jobs that are actively running.
|TCS
|-
|scinet gpc priority
|
|Compute usage in the last 14 days, and how this affects your priority in the queue.
|GPC
|-
|nodeperf
|[''userName'']
|Who is doing what on the current node?
|GPC
|-
|jobperf
| <''jobId''<nowiki> | </nowiki>''jobName''>
|Informs about the performance of all nodes of a given job.
|GPC
|-
|jobError
| <''jobId''<nowiki> | </nowiki>''jobName''>
| Displays realtime error output of a given job.
|GPC
|-
|jobOutput
| <''jobId''<nowiki> | </nowiki>''jobName''>
| Displays realtime standard output of a given job.
|GPC
|-
|jobcd
| <''jobId''<nowiki> | </nowiki>''jobName''>
| Allows users to quickly move into the working directory of a given job.
|GPC
|-
|jobscript
| <''jobId''<nowiki> | </nowiki>''jobName''>
| Displays the submission script used when submitting a given job.
|GPC
|-
|jobssh
| <''jobId''<nowiki> | </nowiki>''jobName''>
| Allows users to connect to the head-node of a given job.
|GPC
|-
|jobtop
| <''jobId''<nowiki> | </nowiki>''jobName''>
| Allows users to "top" on the head-node of a given job.
|GPC
|-
|jobdep
|<''jobId''>
|Displays dependencies of given job.
|GPC
|-
|jobtree
|[''userName'']
|Displays the jobs tree of dependencies for a given user.
|GPC
|-
|debugjob
|[''nodes'']
|Requests a time-limited interactive session of up to 8 dedicated nodes
|GPC
|-
|}

= Modules =
{| border="1" cellpadding="5" cellspacing="0"
!{{Hl2}} | Command
!{{Hl2}}| Arguments
!{{Hl2}}| Description
!{{Hl2}}| Cluster
|-
|modulefind
|[''searchWord'']
|Finds modules containing ''searchWord'' and indicate their compatibility with already loaded modules.
|GPC
|-
|moduleadvice
|<''modulename''>
|Show the module commands required to load ''modulename'', taking into account prerequisites and conflicts with already loaded modules.
|GPC
|-
|modulewrap
|''<comma-separated-modules>'' ''<command>''
|Load the modules in the ''comma-separated-modules'' list (no spaces!) only to run the given ''command''.
|GPC
|-
|}

SciNet Usage Reports

2015-04-30T14:49:18Z

Nolta:

SciNet usage reports are now available on the CCDB website -
https://ccdb.computecanada.ca/. Login and then click on "View Group Usage"
under the "My Account" menu.
Please let us know if you have comments/questions about these reports.

PIs and all people that they sponsor will be able to view the reports
for only their group.

There is a summary report which gives a high-level
overview for those groups with RAC allocations as well as more
detailed cumulative reports which breakdown usage by allocation, system
and user. These are all updated nightly.

Oldwiki.scinet.utoronto.ca:System Alerts

2015-04-22T18:25:16Z

Nolta: bgq up

Oldwiki.scinet.utoronto.ca:System Alerts

2015-04-22T17:44:20Z

Nolta: bgq down

SciNet User Support Library

2015-03-18T18:25:27Z

Nolta: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* Mar 18, BGQ: module hdf5/1812-v18-mpich2-gcc is deprecated; please use hdf5/1814-v18-mpich2-gcc instead.

* Mar 13, BGQ: module fftw/3.3.3-gcc4.8.1 is deprecated; please use fftw/3.3.4-gcc4.8.1 instead.

* Mar 05, BGQ: modules bgqgcc/4.8.1 and mpich2/gcc-4.8.1 now fixed for V1R2M2.

* Mar 04, GPC: OpenBLAS 0.2.13 installed as modules.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[TCS_Quickstart|TCS: Tightly Coupled System]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[GPU_Devel_Nodes|ARC: GPU nodes]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [[File:Rss.gif|link=http://support.scinet.utoronto.ca/podcasts.xml]] [http://support.scinet.utoronto.ca/podcasts.xml SciNet Training Webcasts]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

SciNet User Support Library

2015-03-18T18:23:27Z

Nolta: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* Mar 18, BGQ: module hdf5/1812-v18-mpich2-gcc is deprecated; please use hdf5/1814-v18-mpich2-gcc instead.

* Mar 05, BGQ: modules bgqgcc/4.8.1 and mpich2/gcc-4.8.1 now fixed for V1R2M2.

* Mar 04, GPC: OpenBLAS 0.2.13 installed as modules.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[TCS_Quickstart|TCS: Tightly Coupled System]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[GPU_Devel_Nodes|ARC: GPU nodes]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [[File:Rss.gif|link=http://support.scinet.utoronto.ca/podcasts.xml]] [http://support.scinet.utoronto.ca/podcasts.xml SciNet Training Webcasts]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

SciNet User Support Library

2015-02-11T14:34:55Z

Nolta: /* System Updates */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System Updates==

* Feb 6, GPC: intel/15.0 module is deprecated -- use 15.0.1 instead.

* Feb 5, GPC, BGQ: git/1.9.5 is available as a module.

* Jan 22, [[BGQ]] now a single 4-rack system. bgqdev-fen1 is the single login/devel/submission node.

* Jan 19, GPC: emacs 24.4 available as a module.

* Jan 14, GPC: kernel upgraded to 2.6.32-504.3.3.el6.x86_64. Its base OS remains unchanged.

* Jan 10, GPC: Cmake 3.1.0 available as a module.

* Jan 7, GPC: ROOT 6.0.02 installed as a module.

* Jan 6, GPC: Ruby 1.9.3 installed as a module.

* Dec 12, [[BGQ]]: devel system upgraded to 2 full racks (32,768 cores).

([[Previous_updates:|Previous updates]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[TCS_Quickstart|TCS: Tightly Coupled System]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[GPU_Devel_Nodes|ARC: GPU nodes]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [[File:Rss.gif|link=http://support.scinet.utoronto.ca/podcasts.xml]] [http://support.scinet.utoronto.ca/podcasts.xml SciNet Training Webcasts]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

SciNet User Support Library

2015-02-06T20:43:33Z

Nolta: git, intel module updates

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System Updates==

* Feb 6, GPC: intel/15.0 is deprecated -- use 15.0.1 instead.

* Feb 5, GPC, BGQ: git/1.9.5 is available as a module.

* Jan 22, [[BGQ]] now a single 4-rack system. bgqdev-fen1 is the single login/devel/submission node.

* Jan 19, GPC: emacs 24.4 available as a module.

* Jan 14, GPC: kernel upgraded to 2.6.32-504.3.3.el6.x86_64. Its base OS remains unchanged.

* Jan 10, GPC: Cmake 3.1.0 available as a module.

* Jan 7, GPC: ROOT 6.0.02 installed as a module.

* Jan 6, GPC: Ruby 1.9.3 installed as a module.

* Dec 12, [[BGQ]]: devel system upgraded to 2 full racks (32,768 cores).

([[Previous_updates:|Previous updates]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[TCS_Quickstart|TCS: Tightly Coupled System]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[GPU_Devel_Nodes|ARC: GPU nodes]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [[File:Rss.gif|link=http://support.scinet.utoronto.ca/podcasts.xml]] [http://support.scinet.utoronto.ca/podcasts.xml SciNet Training Webcasts]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

BGQ

2015-02-05T20:48:58Z

Nolta: add git module

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=Aug 2012, Nov 2014
|operatingsystem= RH6.3, CNK (Linux)
|loginnode= bgqdev-fen1
|nnodes= 4096 nodes (65,536 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP & LKSAVI==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

A half-rack of BlueGene/Q (8,192 cores) was purchased by the [http://likashingvirology.med.ualberta.ca/ Li Ka Shing Institute of Virology] at the University of Alberta in late fall 2014 and integrated into the existing BGQ system.

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet's BGQ consists of 8 mdiplanes (four-racks) totalling 65,536 cores and 64TB of RAM.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|-
| 96 (3 racks)
| 3072
| 49152
| 4x4x12x8x2
|-
| 128 (4 racks)
| 4096
| 65536
| 8x4x8x8x2
|}

== Login/Devel Node ==

The development node is '''bgqdev-fen1''' which one can login to from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development node is a Power7 machines running Linux which serve as the compilation and submission host for the BGQ. Programs are cross-compiled for the BGQ on this node and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit is 24 hours. Official SOSCIP project jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>
=== Steps ( Job dependency) ===
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step1
# @ queue
# Launch the first step :
if [ $LOADL_STEP_NAME = "step1" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step2
# @ dependency = step1 == 0
# @ queue
# Launch the second step if the first one has returned 0 (done successfully) :
if [ $LOADL_STEP_NAME = "step2" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step3
# @ dependency = step2 == 0
# @ queue
# Launch the third step if the second one has returned 0 (done successfully) :
if [ $LOADL_STEP_NAME = "step3" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Note: when running a job under ddt, you'll need to add "<tt>--ranks-per-node=X</tt>" to the "runjob arguments" field.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
The BGQ GPFS file system, except for HPSS, is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers & Development Tools'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Git
| 1.9.5
| Revision control system
| <tt>git, gitk</tt>
| <tt>git</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd/2.9-smp</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

Software and Libraries

2015-02-05T20:45:42Z

Nolta: add git 1.9.5

= Software Module System =
All the software listed on this page is accessed using a modules system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.

Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.

{|
!{{Hl2}}|Function
!{{Hl2}}|Command
!{{Hl2}}|Comments
|-
|List available software packages:
|<pre>$ module avail</pre>
|
*If a module is not listed here, it is not supported.
*The flag "(default)" is never part of the name.
|-
|Use particular software:
|<pre> $ module load [module-name] </pre>
|
*If possible, specify only the short name (the part before the "/").
*When ambiguous, this loads the default one.
|-
|List available versions of a specific software package:
|<pre>$ module avail [short-module-name]</pre>
|
|-
|List currently loaded modules:
|<pre>$ module list</pre>
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.
|-
|Get description of a particular module:
|<pre>$ module help [module-name]</pre>
|
|-
|Remove a module from your shell:
|<pre>$ module unload [module-name]</pre>
|
|-
|Remove all modules:
|<pre>$ module purge</pre>
|
|-
|Replace one loaded module with another:
|<pre>$ module switch [old-module-name] [new-module-name]</pre>
|
|}
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[short-module-name]_BASE
SCINET_[short-module-name]_LIB
SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Errors in loaded modules can arise for a few reasons, for instance:
* A module by that name may not exist.
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.

It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.

== Default and non-default modules ==

When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.

== Naming convention ==

For modules that access applications, the full name of a module is as follows.
<pre>
[short-module-name]/[version-number]
</pre>

To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:
{|border="1" cellspacing="0" cellpadding="2"
| '''old name'''
| '''new name'''
| '''remarks'''
|-
|autoconf/autoconf-2.64     
|autoconf/2.64
|''short name unchanged''
|-
|cuda/cuda-3.0
|cuda/3.0
|''default's short name unchanged''
|-
|cuda/cuda-3.1
|cuda/3.1
|
|-
|debuggers/ddd-3.3.12
|ddd/3.3.12
|
|-
|debuggers/gdb-7.1
|gdb/7.1
|
|-
|editors/nano/2.2.4
|nano/2.2.4
|
|-
|emacs/emacs-23.1
|emacs/23.1.1
|''short name unchanged''
|-
|gcc/gcc-4.4.0
|gcc/4.4.0
|''short name unchanged''
|-
|graphics/ncview
|ncview/1.93
|
|-
|graphics/graphics
|grace/5.1.22
|
|-
|
|gnuplot/4.2.6
|
|-
|svn/svn165
|svn/1.6.5
|''short name unchanged''
|-
|visualization/paraview
|paraview/3.8
|
|-
|amber10/amber10
|amber/10.0.30
|
|-
|gamess/gamess
|gamess/May2209  
|''default's short name unchanged''
|}

==modulefind - Finding modules by name==

The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.

A little SciNet utility called <tt>modulefind</tt> (one word) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have
been loaded, could be loaded, cannot because of conflicts with
already loaded modules, or have unresolved dependencies
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.

Note that just 'modulefind' will list all top-level modules.

== Making your own modules ==

How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.

== Deprecated modules ==

Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre>

Currently (Oct 5,2010), the following modules are deprecated on the GPC:
<pre>
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025
</pre>

On the TCS, currently (Oct 5,2010) the only deprecated module is:
<pre>
ncl/5.1.1old
</pre>

Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.

== Commercial software ==

Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc.
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== Software lists ==
=== ARC/GPU Software ===

The CPUs in the GPU nodes of the ARC cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.

=== GPC Software ===

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Software
!{{Hl2}}| Versions
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|Intel Compiler
|12.1.3*, 12.1.5, 13.1.1, 14.0.1, 15.0, 15.0.1
| includes MKL library, which includes BLAS, LAPACK, FFT, ...
| <tt>icpc,icc,ifort</tt>
| <tt>intel</tt>
|-
| GCC Compiler
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2, 4.8.1, 4.9.0
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Cuda
| 3.2, 4.0, 4.1*, 4.2, 5.0, 5.5, 6.0
| NVIDIA's extension to C for GPGPU programming
| <tt>nvcc</tt>
| <tt>cuda</tt>
|-
| PGI Compiler
| 12.5, 12.6*, 13.2
| supports OpenACC and CUDA Fortran
| <tt>pgcc,pgcpp,pgfortran</tt>
| <tt>pgi</tt>
|-
| IntelMPI
| 4.0.2, 4.0.3*, 4.1.2, 5.0.1, 5.0.2
| MPICH2 based MPI
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|-
| OpenMPI
| 1.4.4*, 1.5.4, 1.6.4, 1.8.3
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|-
| UPC
| 2.12.2
| Berkley Unified Parallel C Implementation
| <tt>upcc</tt>
| <tt>upc</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''
|-
| Nano
| 2.2.4
| Nano's another editor
| <tt>nano</tt>
| <tt>nano</tt>
|-
| Emacs
| 23.1.1, 24.4*
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|-
| XEmacs
| 21.4.22
| XEmacs editor
| <tt>xemacs</tt>
| <tt>xemacs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''
|-
| Autoconf
| 2.68
|
| <tt>autoconf, ...</tt>
| <tt>autoconf</tt>
|-
| Automake
| 1.11.2
|
| <tt>aclocal, automake</tt>
| <tt>automake</tt>
|-
| CMake
| 2.8.6, 2.8.8*, 3.1.0
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Scons
| 2.0
| Software construction tool
| <tt>scons</tt>
| <tt>scons</tt>
|-
| Git
| 1.7.1, 1.7.10*, 1.9.5
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|-
| Intel tools
| 2013, 2015
| Intel Code Analysis Tools
| Vtune Amplifier XE, Inspector XE
| <tt>inteltools</tt>
|-
| Mercurial
| 1.8.2
| Version control system (part of the python module!)
| <tt>hg</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1, 4.2.1*
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler
| <tt>ddt, map</tt>
| <tt>ddt</tt>
|-
| DDD
| 3.3.12
| Data Display Debugger
| <tt>ddd</tt>
| <tt>ddd</tt>
|-
| GDB
| 7.3.1*, 7.6
| GNU debugger (the intel idbc debugger is available by default)
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| MPE2
| 2.4.5
| Multi-Processing Environment with intel + OpenMPI
| <tt>mpecc, mpefc, jumpshot</tt>
| <tt>mpe</tt>
|-
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]
| 2.1
| sampling and MPI tracing
| <tt>openss, ...</tt>
| <tt>openspeedshop</tt>
|-
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]
| 1.3.3, 1.4.3*
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)
| <tt>scalasca</tt>
| <tt>scalasca</tt>
|-
| [http://ipm-hpc.sourceforge.net IPM]
| 0.983
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]
| <tt>ipm, ipm_parse, ploticus,...</tt>
| <tt>ipm</tt>
|-
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]
| 3.7,0*, 3.9.0
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|-
| Padb
| 3.2
| examine and debug parallel programs
| <tt>padb</tt>
| <tt>padb</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Visualization tools'''''
|-
| Grace
| 5.1.22
| Plotting utility
| <tt>xmgrace</tt>
| <tt>grace</tt>
|-
| Gnuplot
| 4.2.6, 4.6.1*
| Plotting utility Requires 'extras' module if used on compute nodes.
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| [[ Using_Paraview | ParaView ]]
| 3.12.0*, 3.14.1, 4.1.0
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>paraview</tt>
|-
| VMD
| 1.8.6, 1.9*
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|-
| NCL/NCARG
| 6.0.0
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|-
| ROOT
| 5.30.00, 5.30.03*, 5.34.03, 6.02.02
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|-
| ImageMagick
| 6.6.7
| Image manipulation tools
| <tt>convert,animate,composite,...</tt>
| <tt>ImageMagick</tt>
|-
| PGPLOT
| 5.2.2
| Graphics subroutine library
| <tt>libcpgplot,libpgplot,libtkpgplot</tt>
| <tt>pgplot</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| NetCDF
| 4.1.3*, 4.2.1, 4.3.2
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|-
| Ncview
| 2.1.1, 2.1.
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>ncview</tt>
|-
| NCO
| 4.0.8, 4.3.2
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|-
| CDO
| 1.5.1, 1.5.4*, 1.6.1
| Climate Data Operators
| <tt>cdo</tt>
| <tt>cdo</tt>
|-
| UDUNITS
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| HDF4
| 4.2.6
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|-
| [[Hdf5 | HDF5]]
| 1.8.7-v18*
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|-
| [http://www.arg0.net/encfs EncFS ]
| 1.74
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)
| <tt>encfs</tt>
| <tt>encfs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| [[gamess|GAMESS (US)]]
| August 18, 2011 R1
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|-
| [[gromacs|GROMACS]]
| 4.5.5, 4.5.7, 4.6.2, 4.6.3
| GROMACS molecular dynamics, single precision, MPI
| <tt>grompp, mdrun</tt>
| <tt>gromacs</tt>
|-
| [[namd|NAMD]]
| 2.8, 2.9*
| NAMD - Scalable Molecular Dynamics
| <tt>namdmpiexec, namd2</tt>
| <tt>namd</tt>
|-
| [[nwchem|NWChem]]
| 6.0
| NWChem Quantum Chemistry
| <tt>nwchem</tt>
| <tt>nwchem</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 4.3.2, 5.0.3
| Quantum Chemistry
| <tt>pw.x, ...</tt>
| <tt>espresso</tt>
|-
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|-
|-
| [http://denovoassembler.sourceforge.net RAY]
| 2.1.0 (small k-mer), 2.2.0, 2.3.1
| Parallel de novo genome assemblies
| <tt>Ray</tt>
| <tt>ray</tt>
|-
| [[cpmd|CPMD]]
| 3.13.2
| Carr-Parinello molecular dynamics, MPI
| <tt>cpmd.x</tt>
| <tt>cpmd</tt>
|-
| [[R Statistical Package|R]]
| 2.13.1, 2.14.1, 2.15.1*, 3.0.0, 3.0.1, 3.1.1
| statistical computing
| <tt>R</tt>
| <tt>R</tt>
|-
| Octave
| 3.4.3*, 3.8.1
| Matlab-like environment
| <tt>octave</tt>
| <tt>octave</tt>
|-
| [http://www.openfoam.org OpenFOAM ]
| 2.1.0, 2.3.0
| Open Source CFD Package
| <tt>*foam</tt>
| <tt>openfoam</tt>
|-
| Bedtools
| 2.21.0
| Toolset for genome arithmetic
| <tt>bedtools ...</tt>
| <tt>bedtools</tt>
|-
| Samtools
| 0.1.19
| Suite of programs for interacting with high-throughput sequencing data
|
| <tt>samtools</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
| 3.1*, 3.2, 3.3, 3.4.4
| Portable, Extensible Toolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|-
| BOOST
| 1.47.0, 1.54, 1.55
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| [[Armadillo C++ linear algebra library | Armadillo]]
| 3.910.0
| C++ armadillo libraries (implement Matlab-like syntax)
| <tt>armadillo</tt>
| <tt>armadillo</tt>
|-
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]
| 1.13
| Optimized BLAS implementation
| <tt>libgoto2</tt>
| <tt>gotoblas</tt>
|-
| OpenBLAS
| 1.13
| Open BLAS implementation including lapack.
| <tt>libopenblas</tt>
| <tt>openblas</tt>
|-
| GSL
| 1.13*, 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| FFTW
| 2.1.5, 3.3.0, 3.3.3*
| fast Fourier transform library
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|-
| LAPACK
|
| Provided by the Intel MKL library
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/
| <tt>intel</tt>
|-
| [http://freshmeat.net/projects/rlog RLog ]
| 1.4
| RLog provides a flexible message logging facility for C++ programs and libraries.
| <tt>librlog</tt>
| <tt>cxxlibraries/rlog</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[GNU Parallel]]
| 2012-10-22, 20140622*
| execute commands in parallel
| <tt>parallel</tt>
| <tt>gnu-parallel</tt>
|-
| [[Python]]
| 2.7.2*, 2.7.3, , 2.7.5, 2.7.8
| Python programming language. See [[Python]] page for details on installed packages.
| <tt>python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.3.4
| Python programming language. Modules included : numpy 1.8.1 , scipy 0.14.0 , matlotlib 1.3.1 , ipython 1.2.1 , cython 0.20.1 , h5py-2.3.0 , tables-3.1.1 , netCDF4-1.1.0 , astropy-0.3.2 , scikit_learn-0.15.0b1
| <tt>python</tt>
| <tt>python</tt>
|-
| Ruby
| 1.9.1*, 1.9.3
| Ruby programming language
| <tt>ruby</tt>
| <tt>ruby</tt>
|-
| Java
| 1.6.0, 1.7.1*
| IBM's Java JRE ad SDK
| <tt>java</tt>, <tt>javac</tt>
| <tt>java</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| Xlibraries
|
| A collection of X graphics libraries and tools
| <tt>xterm</tt>, <tt>xpdf</tt>, ...
| <tt>Xlibraries</tt>
|-
| Extras
|
| A collection of standard linux and home-grown tools
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...
| <tt>extras</tt>
|}

''* Several versions of this module are installed; listed is the default version.''

=== TCS Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM compilers
|10.1(c/c++) 12.1(fortran)
| See [[TCS Quickstart]]
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt>
| ''standard available''
|-
|IBM MPI library
|
| See [[TCS Quickstart]]
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt>
| ''standard available''
|-
| UPC
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
|IBM fortran compiler
|13.1, 14.1
| newer version
| <tt>xlf,xlf_r</tt>
| xlf/13.1
|-
|IBM c/c++ compilers
|11.1, 12.1
| new versions
| <tt>xlc,xlC,xlc_r,xlC_r</tt>
| vacpp
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1, 4.2.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| MPE2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| Scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| HDF5
|
| Scientific data storage and retrieval, parallel I/O Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libhdf5</tt>
| <tt>extras</tt>
|-
| NetCDF + ncview
| 4.0.1*
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.1.1*
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 3.9.6*
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| FFTW
| 3.2.2
| Fast Fourier transform library Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi,libfftw3</tt>
| <tt>extras</tt>
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| LAPACK+SCALAPACK
| 3.4.2+2.0.2
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.
| <tt>liblapack,libscalapack,libblacs</tt>
| <tt>lapack</tt>
|-
| PetSc
| 3.2
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.
| <tt>libpetsc,...</tt>
| <tt>petsc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gmake
| 3.82
| GNU's make. Replaces AIX make or gmake 3.80.
| <tt>make</tt>, <tt>gmake</tt>
| <tt>gmake</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
|}
''* Several versions of this module are installed; listed is the default version.''

=== P7 Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1, 13.1
|See [[P7 Linux Cluster]]
|<tt>xlf,xlf_r,xlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1, 11.1
|See [[P7 Linux Cluster]]
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt>
|vacpp
|-
| Binutils
| 2.23.2
|
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
|IBM MPI library
|5.2.2
|IBM's Parallel Environment
|<tt>mpcc,mpCC,mpfort,mpiexec</tt>
|pe
|-
| GCC Compiler
| 4.6.1 , 4.8.1
| GNU Compiler Collection
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Java
| 7.0
| IBM Java 1.7 implementation
| <tt>javac</tt>
| <tt>jdk</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1, 4.2.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF5
| 1.8.7
| Scientific data storage and retrieval, parallel I/O
| <tt>libhdf5</tt>
| <tt>hdf5</tt>
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.2.0
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.7.5
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gnuplot
| 4.6.1
| command-driven interactive function and data plotting program
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| extras
|
| Adds paths to a fuller set of applications and libraries to your user environment
| <tt>bindlaunch, ...</tt>
| <tt>extras</tt>
|}

=Manuals=
{{:Manuals}}

BGQ

2015-02-04T20:18:09Z

Nolta: small clarification for running jobs under ddt

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=Aug 2012, Nov 2014
|operatingsystem= RH6.3, CNK (Linux)
|loginnode= bgqdev-fen1
|nnodes= 4096 nodes (65,536 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP & LKSAVI==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

A half-rack of BlueGene/Q (8,192 cores) was purchased by the [http://likashingvirology.med.ualberta.ca/ Li Ka Shing Institute of Virology] at the University of Alberta in late fall 2014 and integrated into the existing BGQ system.

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet's BGQ consists of 8 mdiplanes (four-racks) totalling 65,536 cores and 64TB of RAM.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|-
| 96 (3 racks)
| 3072
| 49152
| 4x4x12x8x2
|-
| 128 (4 racks)
| 4096
| 65536
| 8x4x8x8x2
|}

== Login/Devel Node ==

The development node is '''bgqdev-fen1''' which one can login to from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development node is a Power7 machines running Linux which serve as the compilation and submission host for the BGQ. Programs are cross-compiled for the BGQ on this node and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit is 24 hours. Official SOSCIP project jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>
=== Steps ( Job dependency) ===
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step1
# @ queue
# Launch the first step :
if [ $LOADL_STEP_NAME = "step1" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step2
# @ dependency = step1 == 0
# @ queue
# Launch the second step if the first one has returned 0 (done successfully) :
if [ $LOADL_STEP_NAME = "step2" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step3
# @ dependency = step2 == 0
# @ queue
# Launch the third step if the second one has returned 0 (done successfully) :
if [ $LOADL_STEP_NAME = "step3" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Note: when running a job under ddt, you'll need to add "<tt>--ranks-per-node=X</tt>" to the "runjob arguments" field.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
The BGQ GPFS file system, except for HPSS, is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd/2.9-smp</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

Scheduler

2015-02-04T16:34:25Z

Nolta: better link to moab command list

The queueing system used at SciNet is based around Cluster Resources [http://www.adaptivecomputing.com/products/hpc-products/moab-hpc-basic-edition/ Moab Workload Manager].
Moab is used on both the GPC and TCS however [http://www.adaptivecomputing.com/products/open-source/torque/ Torque] is used as the backend resource manager on the GPC and IBM's [http://www.ibm.com/software/products/en/tivoliworkloadschedulerloadleveler/ LoadLeveler] is used on the TCS.

This page outlines some of the most common Moab commands with full documentation available from Moab [http://docs.adaptivecomputing.com/mwm/7-2-9/help.htm#a.gcommandoverview.html%3FTocPath%3D4.0%2520Scheduler%2520Commands|_____0 here], the torque (pbs) commands full documentation is [http://docs.adaptivecomputing.com/torque/4-2-9/help.htm here].

Some common questions about the queuing system can be found on the [[FAQ]] as well.

=== Queues ===

==== GPC ====

===== batch =====

The batch queue is the default queue on the GPC allowing the user access to all the
resources for jobs between 15 minutes and upto 48 hours. If a specific queue is not specified, <tt>-q</tt> flag,
then a job is submitted to the batch queue. Most jobs will run in batch, and user's can use
feature flags to determine the type of nodes they require.

For example, to request two nodes anywhere on the GPC (QDR or DDR), use
<pre>
#PBS -l nodes=2:ppn=8,walltime=1:00:00
</pre>

For two nodes using DDR, use
<pre>
#PBS -l nodes=2:ddr:ppn=8,walltime=1:00:00
</pre>

To get two nodes using QDR, instead, you would use
<pre>
#PBS -l nodes=2:qdr:ppn=8,walltime=1:00:00
</pre>

===== debug =====

A debug queue has been set up primarily for code developers to quickly test
and evaluate their codes and configurations without having to wait in the batch queue. There are 10 nodes
currently reserved for the debug queue. It has quite restrictive limits to promote high turnover
and availability thus a user can only use 2 nodes (16 cores) for 2 hours, to a maximum
of 8 nodes (64 cores) for 1/2 an hour and can only have one job in the debug queue at a time. There
is no minimum time limit on this queue.

<pre>
$ qsub -l nodes=1:ppn=8,walltime=1:00:00 -q debug -I
</pre>

===== largemem =====

The largemem queue is used for accessing one of two 16 core with 128 GB memory intel Xeon (non-nehalem) nodes.

<pre>
$ qsub -l nodes=1:ppn=16,walltime=1:00:00 -q largemem -I
</pre>

==== TCS ====

The TCS currently only has one queue, or class, in use called "verylong" and its only
limitation is that jobs must be under 48 hours.

<pre>
#@ class = verylong
</pre>

=== Job Info===

To see all jobs queued on a system use
<pre>
$ showq
</pre>

Three sections are shown; running, idle, and blocked. Idle jobs are commonly referred to as queued jobs
as they meet all the requirements, however they are waiting for available resources. Blocked jobs
are either caused by improper resource requests or more commonly by exceeding a user or groups allowable
resources. For example if you are allowed to submit 10 jobs and you submit 20, the first 10
jobs will be submitted properly and either run right away or be queued, however the other 10 jobs
will be blocked and the jobs won't be submitted to the queue until one of the first 10 finishes.

=== Available Resources ===

Determining when your job will run can be tricky as it involves a combination of queue type, node type, system reservations, and job priority. The following commands are provided to help you figure out what resources are currently available, however they may not tell you exactly when your job will run for the aforementioned reasons.

==== GPC ====
To show how many qdr nodes are currently free, use the show back fill command
<pre>
$ showbf -f qdr
</pre>

To show how many ddr nodes are free, use
<pre>
$ showbf -f ddr
</pre>

==== TCS ====
To show how many TCS nodes are free, use
<pre>
$ showbf -c verylong
</pre>

For example checking for a qdr job

<pre>
$ showbf -f qdr
Partition Tasks Nodes Duration StartOffset StartDate
--------- ----- ----- ------------ ------------ --------------
ALL 14728 1839 7:36:23 00:00:00 00:23:37_09/24
ALL 256 30 INFINITY 00:00:00 00:23:37_09/24
</pre>

shows that for jobs under 7:36:23 you can use 1839 nodes, but if you submit
a job over that time only 30 will be available. In this case this is
due to a large reservation made my SciNet staff, but from a users point
of view, showbf tells you very simply what is available and at what time point.
In this case, a user may wish to set #PBS -l walltime=7:30:00 in their script, or add -l walltime=7:30:00 to their qsub command in order to ensure that the jobs backfill the reserved nodes.

'''NOTE:''' showbf shows currently available nodes, however just because nodes are available
doesn't mean that your job will start right away. Job priority, system reservations
along with dedicated nodes, such as those for the debug queue, will alter when jobs
run so even if enough nodes appear "free", it doesn't mean your job will actually run right
away.

=== Job Submission ===

==== Interactive ====

On the GPC an interactive queue session can be requested using the following
<pre>
$ qsub -l nodes=2:ppn=8,walltime=1:00:00 -I
</pre>

In case you may experience longer than usual delays when requesting an interactive session, try specifying the "debug" queue:
<pre>
$ qsub -l nodes=2:ppn=8,walltime=1:00:00 -I -q debug
</pre>

==== Non-interactive (Batch) ====

For a non-interactive job submission you require a submission script formatted for the appropriate resource manger. Examples
are provided for the [[GPC_Quickstart#Submitting_A_Batch_Job | GPC]] and [[TCS_Quickstart#Submitting_A_Job | TCS]].

=== Job Status ===
<pre>
$ qstat jobid
</pre>
To see the status of all your jobs, use
<pre>
showq -u username
</pre>
or
<pre>
qstat -u username
</pre>

If your job appears to be blocked, you can try the following command
<pre>
$ checkjob jobid
</pre>
which gives more verbose and often a bit confusing output.

=== Cancel a Job ===

<pre>
$ canceljob jobid
</pre>

=== Accounting ===

For any user with an NRAC/LRAC allocation, a special account with the Resource Allocation Project (RAP) identifier (RAPI) from Compute Canada Database (CCDB) is set up in order to access the allocated resources. Please use the following instructions to run your job using your special allocation. This is necessary both for accounting purposes as well as to assign the appropriate priority to your jobs.

Each job run on the system will have a default RAP associated with it. Most users already have their default RAP properly set. However, if you have more than one allocation (different RAPs), you may need/want to change your default RAP in order to charge your jobs to a particular RAP.

==== Changing your default RAP ====

# Go to the [https://portal.scinet.utoronto.ca portal], login with your SciNet username and password.
# Click on "Change SciNet default RAP" and change your default RAP.

==== Specifying the RAP for GPC ====

Alternatively, you may want to assign a RAP for each particular job you run. There are two ways to specify an account for Moab/Torque: From the command line or inside the batch submission script.

===== Command line =====

Use the '-A RAPI' flag when you submit your job using qsub. Note that the command line option will override the submission script if an account is specified on both the submission script and the command line. "RAPI" is the RAP Identifier, e.g. abc-123-de.

===== Submission Script =====

Add a line in your submit script as follows:
<pre>
#PBS -A RAPI
</pre>

Please replace "RAPI" with your RAP Identifier.

==== Specifiying the RAP for TCS ====

Add a line in your submit script as follows:
<pre>
# @ account_no = RAPI
</pre>

Please replace "RAPI" with your RAP Identifier.

=== User Stats ===

Show current usage stats for a $USER

<pre>
$ showstats -u $USER
</pre>

=== Reservations ===
<pre>
$ showres
</pre>

Standard users can only see their reservations not other users or system ones.
To determine what is available a user can use "showbf", it shows what resources are
available and at what time level, taking into account running jobs and all the reservations. Refer to the [[Moab#Available_Resources | Available Resources]] section of this page for more details.

=== Job Dependencies ===

Sometimes you may want one job not to start until another job finishes, however
you would like to submit them both at the same time. This can be done
using job dependencies on both the GPC and TCS, however the commands
are different due to the underlying resource managers being different.

==== GPC ====

Use the -W flag with the following syntax in your submission script to have this job not start
until the job with jobid or jobName (given with -N jobName) has successfully finished
<pre>
-W depend=afterok:{jobid | jobName}
</pre>

More detailed syntax and examples can be found
[[http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#-W here]].

==== TCS ====

Loadleveler does job dependencies using what they call steps.
See the [[TCS_Quickstart#Steps | TCS Quickstart]] guide for an example.

=== Adjusting Job Priority ===

The ability to adjust job priorities downwards can also be of use to adjust relative priorities of jobs between users who are running jobs of the same allocation (eg, a default or NRAC allocation of the same PI). Priorities are determined by how much of the time of that allocation been currently used, and all users using that account will have identical priorities. This mechanism allows users to voluntarily reduce their priority to allow other users of the same allocation to run ahead of them.

In principle, by adjusting a jobs priority downwards, you could reduce your jobs priority to the point that someone elses job entirely could go ahead of yours. In practice, however, this is extremely unlikely. Users with NRAC allocations have priorities that are extremely large positive numbers that depend on their allocation and how much of it they have already used during the past fairshare window (2 weeks); it is very unlikely that two groups would have priorities that are within 10 or 100 or 1000 of each other.

Note that at the moment, we do not allow priorities to go negative; they are integers that can go no lower than 1. (This may change in the future) That means that users of accounts that have already used their full allocation during the current fairshare period (eg, over the past two weeks), and so whose priority would normally be negative but is capped at 1, can not lower their priority any further. Similar, users with a `default' allocation have priority 1, and cannot lower their priorities any further.

==== GPC ====

Moab allows users to adjust their jobs' priority moderately downwards, with the <tt>-p</tt> flag; that is, on a qsub line
<pre>
$ qsub ... -p -10 JOBID
</pre>
or in a script
<pre>
...
#PBS -p -10
..
</pre>

The number used (-10 in the examples above) can be any negative number down to -1024.

The ability to adjust job priorities downwards can be useful when you are running a number of jobs and want some to enter the queue at higher priorities than others. Note that if you absolutely require some jobs to start before others, you could use [[#Job Dependencies | job dependencies]] instead.

For a job that is currently queued, one can adjust its priority with
<pre>
$ qalter -p -10 JOBID
</pre>



=== Suspending a Running Job ===

Separate from, and in addition to, the ability to place a hold on a queued job, you may want to suspend a running job. For example, you may want to test the timing of events in a weakly coupled parallel environment.

==== GPC ====

To suspend a job:

<pre>
qsig -s STOP <jobid>
</pre>

and to start it again:

<pre>
qsig -s CONT <jobid>.
</pre>

Scripts are suspendable by default, so you don't need to add any signal handling for this to work.
As far as we can tell, the result is identical to using fg and ctrl-Z (or kill -STOP <PID>) in an interactive run.

More about using (and trapping) signals can be found on the [[Using Signals]] page.

=== QDR Network Switch Affinity ===

The QDR network is globally 5:1 oversubscribed, but on switch it has full 1:1 cross-section whereas the DDR is completely 1:1 non-blocking.
When a job is submitted to the GPC QDR nodes, the queuing system tries to fulfill the requirements with nodes on the same switch to improve network
performance. If not enough nodes are available on the same switch to satisfy the job request, the queue will then use any available nodes. This behavior can be changed
by using the submission flag " -l nodesetisoptional=false " which forces the queuing system to only
run this job on the same switch, thus the job will stay queued until enough nodes on one switch are available
to satisfy this request. Note that the maximum number of nodes on one switch is 30, so
a request of greater than 30 nodes with this flag will never run.
<pre>
qsub -l nodes=4:ppn=8,walltime=10:00,nodesetisoptional=false
</pre>

=== Multiple Job Submissions ===

If you are doing doing batch processing of a number of similar jobs on the GPC, torque has a feature called job arrays that can be used to simplify this process.
By using the "-t 0-N" option on the command line during job submission or putting it in the job script file, #PBS -t 0-N, torque will expand your
single job submission into N+1 jobs and sets the environment variable PBS_ARRAYID equal to that jobs specific number, ie 0-N, for each job. This
reduces the amount of calls to qsub, and can allow the user to have many less submission scripts. Job arrays also have the benefit of batching
groups of jobs allowing commands like qalter, qdel, qhold to work on all or a subset of the job array jobs with one command, instead of having
to run the command for each job.

In the following example, 10 jobs are submitted using a single command
<pre>
qsub -t 0-9 jobscript.sh
</pre>

and the submission script then modifies the job based on the PBS_ARRAYID.
<pre>
#!/bin/bash
#PBS -l nodes=1:ppn=8,walltime=10:00:00
#PBS -N array_jobs

cd ${PBS_O_WORKDIR}
mkdir job.${PBS_ARRAYID}
cd job.${PBS_ARRAYID}

echo "Running job ${PBS_ARRAYID}"
mpirun -np 8 ./mycode >& array_job.${PBS_ARRAYID}.out
</pre>

The JOBID and the job name both get the additional ARRAYID added onto them in the form of a hyphen, ie JOBID-ARRAYID. If for example you wanted
to cancel all the jobs in a job array you would use "qdel JOBID", whereas if you wanted to cancel just one of the jobs you would use
"qdel JOBID-ARRAYID".

See [http://docs.adaptivecomputing.com/torque/4-1-3/Content/topics/2-jobs/multiJobSubmission.htm here] and [http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm here]
for full details.