https://oldwiki.scinet.utoronto.ca/api.php?action=feedcontributions&feedformat=atom&user=Rzonoldwiki.scinet.utoronto.ca - User contributions [en-gb]2026-05-08T02:53:14ZUser contributionsMediaWiki 1.35.12https://oldwiki.scinet.utoronto.ca/index.php?title=Using_Paraview&diff=9421Using Paraview2018-09-11T18:38:42Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
[http://www.paraview.org/ ParaView] is a powerful, parallel, client-server based visualization system that allows you to use SciNet's GPC nodes to render data on SciNet, and manipulate the results interactively on your own desktop. To use the paraview server on SciNet is much like using it locally, but there is an additional step in setting up a connection directly between your desktop and the compute nodes.<br />
<br />
[[Image:Paraview.png|thumb|right|320px|The ParaView Client GUI]]<br />
<br />
===Installing ParaView===<br />
<br />
To use Paraview, you will have to have the client software installed on your system; you will need ParaView from [http://www.paraview.org/paraview/resources/software.html the Paraview website]. Binaries exist for Linux, Mac, and Windows systems. The client version must exactly match the version installed on the server, currently 3.12 or 3.14.1. The client version has all the functionality of the server, and can analyze data locally.<br />
<br />
===SSH Forwarding For ParaView===<br />
<br />
To interactively use the ParaView server on GPC, you will have to work some ssh magic to allow the client on your desktop to connect to the server through the scinet login nodes. The steps required are<br />
<br />
* Have an SSH key that you can use to log into SciNet<br />
* Submit an interactive job, with a shell on the head node that you'll be running the server on<br />
* Start ssh forwarding<br />
* Start paraview server<br />
* Connecting client and server<br />
<br />
====SSH Keys====<br />
<br />
To be able to log into the compute nodes where ParaView will be running, you'll have to have an [[Ssh_keys | SSH key]] set up, as password authentication won't work. Our [[Ssh_keys | SSH Keys and SciNet]] page describes how to do this.<br />
<br />
====Log into node====<br />
<br />
The first thing to do is to go to the node from which you'll start the ParaView server. This is typically done by starting an interactive job on the GPC, perhaps on the [[Moab#debug | debug ]] queue or sandybridge [[GPC_Quickstart#Memory_Configuration | large memory]] nodes. Paraview can in principle make use of as many nodes as you throw at it. So one might begin jobs as below:<br />
<br />
<pre><br />
qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy -I<br />
</pre><br />
or<br />
<pre><br />
qsub -l nodes=2:ppn=8,walltime=1:00:00 -q debug -I<br />
</pre><br />
<br />
Once this job has started, you'll be placed in a shell on the head node of the job; typing `<pre>hostname</pre>' will tell you the name of the host, eg<br />
<pre><br />
$ hostname<br />
gpc-f148n089-ib0<br />
</pre><br />
or <br />
<pre><br />
$ hostname<br />
gpc-f107n045-ib0<br />
</pre><br />
you will need this hostname in the following steps.<br />
<br />
====Start SSH port forwarding====<br />
<br />
Once the ssh configuration is set, the port forwarding can be started with the command (on your local machine in a terminal window), using the local host name from above - here we'll take the example of gpc-f148n089-ib0:<br />
<pre><br />
$ export gpcnode="gpc-f148n089-ib0"<br />
$ ssh -N -L 20080:${gpcnode}:22 -L 20090:${gpcnode}:11111 login.scinet.utoronto.ca<br />
</pre><br />
this command will not return anything until the forwarding is terminated, and will just look like it's sitting there. It doesn't start a remote shell or command (-N), but it will connect to login.scinet.utoronto.ca, and from there it will redirect your local (-L) port 20080 to <tt>${gpcnode}</tt> port 22, and similarly local port 20090 to <tt>${gpcnode}</tt> port 11111. We'll use the first for ssh'ing to the remote node (mainly for testing), and the second to conect the local paraview client to the remote paraview server.<br />
<br />
To make sure the port forwarding is working correctly, in another window try sshing directly to the compute node from your desktop:<br />
<pre><br />
$ ssh -p 20080 [your-scinet-username]@localhost<br />
</pre><br />
and this should land you directly on the compute node. If it does not, then something is wrong with the ssh forwarding.<br />
<br />
====Start Server====<br />
<br />
Now that the tunnel is set up, on the compute node you can start the paraview server. To do this, you will have to have the following modules loaded:<br />
<br />
<pre><br />
$ module load Xlibraries intel gcc python openmpi paraview<br />
</pre><br />
(You can replace intelmpi with openmpi, and of course any module that is already loaded does have to be loaded again.)<br />
<br />
Then start the paraview server with the intel mpirun as with any MPI job:<br />
<pre><br />
$ mpirun -np [NP] pvserver --use-offscreen-rendering<br />
</pre><br />
where NP is the number of processors; 16 processors per node on the largemem nodes, or 8 per node otherwise. <br />
<br />
Once running, the ParaView server should output<br />
<pre><br />
Listen on port: 11111<br />
Waiting for client...<br />
</pre><br />
<br />
====Connect Client and Server====<br />
<br />
[[Image:Configure.png|thumb|right|320px|Configuring the client]]<br />
<br />
Once the server is running, you can connect the client. Start the ParaView client on your desktop, and choose File->Connect. Click `Add Server', give the server a name (say, GPC), and give the port number 20090. The other values should be correct by default; host is <tt>localhost</tt>, and the server type is Client/Server. Click `Configure'.<br />
<br />
On the next window, you'll be asked for a command to start up the server; select `Manual', and ok.<br />
<br />
Once the server is selected, click `Connect'. On the compute node, the server should respond `Client connected'. In the client window, when you (for instance) select File->Open, you will be seeing the files on the GPC, rather than the local host.<br />
<br />
From here, the [http://paraview.org/Wiki/ParaView ParaView Wiki] can give you instructions as to how to plot your data.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Ssh_keys&diff=9420Ssh keys2018-08-31T23:42:14Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
[[Ssh | SSH]] has an alternative to passwords to authenticate your login; you can generate a key file on a trusted machine and tell a remote machine to trust logins from a machine that presents that key. This can be both convenient and secure, and may be necessary for some tasks (such as connecting directly to compute nodes to use [[Using_Paraview | some visualization packages]]). Here we describe how to setup keys for logging into SciNet.<br />
<br />
==SSH Keys and SciNet==<br />
<br />
[[Ssh | SSH]] is a secure protocol for logging into or copying data to/from remote machines. In addition to using passwords to [http://en.wikipedia.org/wiki/Authentication authenticate] users, one can use cryptographically secure keys to guarantee that a login request is coming from a trusted account on a remote machine, and automatically allow such requests. Done properly, this is as secure as requiring a password, but can be more convenient, and is necessary for some operations.<br />
<br />
On this page, we will assume you are using Linux, Mac OS X, or a similar environment such as [http://www.cygwin.com/ Cygwin] under Windows. If not, the steps will be the same, but how they are done (for instance, generating keys) may differ; look up the documentation for your ssh package for details.<br />
<br />
==Using SSH keys==<br />
===How SSH keys work===<br />
<br />
SSH relies on [http://en.wikipedia.org/wiki/Public-key_cryptography public key cryptography] for its encryption. These cryptosystems have a private key, which must be kept secret, and a public key, which may be disseminated freely. In these systems, anyone may use the public key to encode a message; but only the owner of the private key can decode the message. This can also be used to verify identities; if someone is claiming to be Alice, the owner of some private key, Bob can send Alice a message encoded with Alice's well-known public key. If the person claiming to be Alice can then tell Bob what the message really was, then that person at the very least has access to Alice's private key.<br />
<br />
To use keys for authentication, we:<br />
* Generate a key pair (Private and Public)<br />
* Copy the public keys to remote sites we wish to be able to login to, and mark it as an authorized key for that system<br />
* Ensure permissions are set properly<br />
* Test.<br />
<br />
===Generating an SSH key pair===<br />
<br />
{|border="1"<br />
|<br />
Note: This describes creating ssh key pairs on '''your''' machine, not on SciNet. On SciNet, you already have key pairs generated, sitting in <tt>${HOME}/.ssh/</tt>, and modifying them is likely to cause problems.<br />
|}<br />
<br />
<br />
The first stage is to create an SSH key pair. On most systems, this is done using the command<br />
<br />
ssh-keygen<br />
<br />
This will prompt you for two pieces of information: where to save the key, and a passphrase for the key. The passphrase is like a password, but rather than letting you in to some particular account, it allows you to use the key you've generated to log into other systems. <br />
<br />
There are a series of possible options to <tt>ssh-keygen</tt> which allow increasingly cryptographically secure keys (by increasing the number of bits used in the key), or by choosing different encryption systems. The defaults are fine, and we won't discuss other options here.<br />
<br />
The default location to save the private key is in <tt>${HOME}/.ssh/id_rsa</tt> (for an RSA key); unless you have some specific reason for placing it elsewhere, use this option. The public key will be <tt>id_rsa.pub</tt> in the same directory.<br />
<br />
Your passphrase can be any string, and of any length. It is best not to make it the same as any of your passwords.<br />
<br />
A sample session of generating a key would go like this:<br />
<br />
$ ssh-keygen<br />
Generating public/private rsa key pair.<br />
Enter file in which to save the key (${HOME}/.ssh/id_rsa): <br />
Enter passphrase (empty for no passphrase): <br />
Enter same passphrase again: <br />
Your identification has been saved in ${HOME}/.ssh/id_rsa.<br />
Your public key has been saved in ${HOME}/.ssh/id_rsa.pub.<br />
The key fingerprint is:<br />
79:8e:36:6a:78:7d:cf:80:94:90:92:0e:74:0b:f1:b7 USERNAME@YOURMACHINE<br />
<br />
====Don't Use Passphraseless Keys!====<br />
<br />
If you do not specify a passphrase, you will have a completely "exposed" private key. '''This is a terrible idea.''' If you then use this key for anything it means that anyone who sits down at your desk, or anyone who borrows or steals your laptop, can login to anywhere you use that key (good guesses could come from just looking at your history) without needing any password, and could do anything they wanted with your account or data. Don't use passphraseless keys.<br />
<br />
We should note that we do, in fact, have one necessary and reasonable exception here -- the keys used within SciNet itself. The SciNet key used for within-scinet operations (you already have one in your account in <tt>~/.ssh/id_rsa</tt>) is passphraseless, for two good reasons. One is that, once you are on one SciNet machine (like the login node), you already have read/write access to all your data; all the nodes mount the same file systems. So there is little to be gained in protecting the SciNet nodes from each other. The second is practical; ssh is used to login to compute nodes to start your compute jobs. You obviously can't be asked to type in a passphrase every time one of your jobs starts; you may not be at your computer at that moment. So passphraseless keys are ok ''within'' a controlled environment; but don't use them for remote access.<br />
<br />
===Copying the Public Key to SciNet (and elsewhere)===<br />
<br />
Now that you have this SSH "identity", you use the public (''not'' the private) key for access to remote machines. The public key must be put as one line in the file <tt>/home/USERNAME/.ssh/authorized_keys</tt>. Do not delete the lines already there, or you may end up with strange problems using SciNet machines.<br />
<br />
You can copy your new public key to the SciNet systems<br />
<pre><br />
scp /home/LOCAL_USERNAME/.ssh/id_rsa.pub SCINET_USERNAME@login.scinet.utoronto.ca:newkey<br />
</pre><br />
Then login to SciNet and copy&paste the contents from <tt>~/newkey</tt> into <tt>~/.ssh/authorized_keys</tt>.<br />
<pre><br />
cat ~/newkey >> ~/.ssh/authorized_keys<br />
</pre><br />
<br />
===<tt>.ssh</tt> Permissions===<br />
<br />
Note that <tt>SSH</tt> is very fussy about file permissions; your <tt>~/.ssh</tt> directory must only be accessible by you, and your various key files must not be writable (or in some cases, readable) by anyone else. Sometimes users accidentally reset file permissions while editing these files, and problems happen. If you look at the <tt>~/.ssh</tt> directory itself, it should not be readable at all by anyone else:<br />
<br />
ls -ld ~/.ssh<br />
drwx------ 2 USERNAME GROUPNAME 7 Aug 9 15:43 /home/USERNAME/.ssh<br />
<br />
and <tt>authorized_keys</tt> must not be writable:<br />
<br />
$ ls -l ~/.ssh/authorized_keys <br />
-rw-r--r-- 1 USERNAME GROUPNAME 1213 May 29 2009 /home/USERNAME/.ssh/authorized_keys<br />
<br />
===Testing Your Key===<br />
<br />
Now you should be able to login to the remote system (say, SciNet):<br />
<br />
$ ssh USERNAME@login.scinet.utoronto.ca<br />
Enter passphrase for key '/home/USERNAME/.ssh/id_rsa': <br />
Last login: Tue Aug 17 11:24:48 2010 from HOMEMACHINE<br />
<br />
===================================================<br />
<br />
This SciNet login node is to be used only as a<br />
gateway to the GPC and TCS.<br />
<br />
[...]<br />
scinet04-$<br />
<br />
If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.<br />
<pre><br />
$ ssh USERNAME@login.scinet.utoronto.ca<br />
<br />
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br />
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @<br />
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br />
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!<br />
Someone could be eavesdropping on you right now (man-in-the-middle<br />
attack)!<br />
It is also possible that the RSA host key has just been changed.<br />
The fingerprint for the RSA key sent by the remote host is<br />
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.<br />
Please contact your system administrator.<br />
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of<br />
this message.<br />
Offending key in /home/<user>/.ssh/known_hosts:3<br />
RSA host key for login.scinet.utoronto.ca <br />
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has<br />
changed and you have requested<br />
</pre><br />
<br />
* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be<br />
restarted with the new passphrase ssh key.<br />
<pre><br />
$ ssh USERNAME@login.scinet.utoronto.ca<br />
<br />
<br />
Agent admitted failure to sign using the key.<br />
</pre><br />
<br />
===(Optional) Using <tt>ssh-agent</tt> to Remember Your Passphrase===<br />
<br />
But now you've just replaced having to type a password for login with having to type a passphrase for your key; what have you gained? <br />
<br />
It turns out that there's an automated way to manage ssh "identities", using the <tt>ssh-agent</tt> command, which should automatically be running on newer Linux or Mac&nbsp;OS&nbsp;X machines. You can add keys to this agent for the duration of your login using the <tt>ssh-add</tt> command:<br />
<br />
$ ssh-add<br />
Enter passphrase for /home/USERNAME/.ssh/id_rsa: <br />
Identity added: /home/USERNAME/.ssh/id_rsa (/home/USERNAME/.ssh/id_rsa)<br />
<br />
and then logins will not require the passphrase, as <tt>ssh-agent</tt> will provide access to the key.<br />
<br />
When you log out of your home computer, the ssh agent will close, and next time you log in, you will have to <tt>ssh-add</tt> your key. You can also set a timeout of (say) an hour by using <tt>ssh-add -t 3600</tt>. This minimizes the number of times you have to type your passphrase, while still maintaining some degree of key security.<br />
<br />
<br />
=== Multiple ssh private keys ===<br />
In quite a few situations its preferred to have ssh keys dedicated to each service, specific role or domain.<br />
<pre><br />
ssh-keygen -t rsa -f ~/.ssh/id_rsa.SciNet -C "Key for SciNet"<br />
ssh-keygen -t rsa -f ~/.ssh/id_rsa.SHARCNET -C "Key for SHARCNET"<br />
ssh-keygen -t rsa -f ~/.ssh/id_rsa.DCS -C "Key for Dept. Of Computer Science"<br />
ssh-keygen -t rsa -f ~/.ssh/id_rsa.CITA -C "Key for CITA"<br />
</pre><br />
<br />
Use different file names for each key. Lets assume that there are 2 keys, ~/.ssh/id_rsa.SciNet and ~/.ssh/id_rsa.SHARCNET. The simple way of making sure each of the keys works all the time is to now create config file for ssh:<br />
<pre><br />
touch ~/.ssh/config<br />
chmod 600 ~/.ssh/config<br />
echo "IdentityFile ~/.ssh/id_rsa.SciNet" >> ~/.ssh/config<br />
echo "IdentityFile ~/.ssh/id_rsa.SHARCNET" >> ~/.ssh/config<br />
</pre><br />
<br />
This would make sure that both keys are always used whenever ssh makes a connection. However, ssh config lets you get down to a much finer level of control on keys and other per-connection setups. The recommendation is to use a key selection based on the Hostname. For example, a ~/.ssh/config that looks like this :<br />
<br />
<pre><br />
Host SciNet<br />
Hostname login.scinet.utoronto.ca<br />
IdentityFile ~/.ssh/id_dsa.SciNet<br />
User pinto<br />
<br />
Host SHARCNET<br />
Hostname sharcnet.ca<br />
IdentityFile ~/.ssh/id_rsa.SHARCNET<br />
User jchong<br />
Port 44787<br />
</pre><br />
<br />
And just login with the shortcut:<br />
<pre><br />
$ ssh SciNet<br />
</pre><br />
<br />
= SSH tunnel =<br />
A more obscure use of ssh is to generate a communication tunnel. As an example, assume you want to access a website running on a remotehost using your localhost, but there is a firewall between the 2 systems blocking every port, except incoming ssh.<br />
<br />
The basic syntax of the ssh command for such a purpose is: <br />
<pre><br />
ssh -f -N -L localport:localhost:remoteport user@remotehost<br />
# -f puts ssh in background<br />
# -N makes it not execute a remote command<br />
</pre><br />
<br />
If the remote website broadcasts on the default port 80, you could do the following:<br />
<pre><br />
ssh -L 8080:localhost:remotehost:80 tunneluser@remotehost<br />
</pre><br />
... and point your local browser to http://localhost:8080<br />
<br />
If you don't want to remember the above sequence of flags all the time, you can add an entry to your ~/.ssh/config:<br />
<pre><br />
Host tunnel<br />
HostName remotehost<br />
IdentityFile ~/.ssh/id_rsa.tunnel<br />
LocalForward 8080 127.0.0.1:80<br />
User tunneluser<br />
</pre><br />
<br />
To open the tunnel just issue the command:<br />
<pre><br />
$ ssh -f -N tunnel<br />
</pre></div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Gamess&diff=9419Gamess2018-08-31T23:41:43Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
===GAMESS(US)===<br />
<br />
====GAMESS version August 18, 2011, R1, on CentOS6====<br />
<br />
The current version of GAMESS(US), August 18, 2011 R1, was built on CentOS6, using the Intel v12.1 compilers and IntelMPI v4.0.2.003 MPI library. The build was done according to the documentation that came with the package, available on the SciNet systems in the /scinet/gpc/src/gamess-Aug1811R1 directory. After running "./config", with the intel and intelmpi modules loaded, the following configuration file (install.info) was obtained:<br />
<br />
[dgruner@gpc-f103n058 gamess-Aug1811R1]$ cat install.info<br />
#!/bin/csh<br />
# compilation configuration for GAMESS<br />
# generated on gpc-f103n058<br />
# generated at Wed Nov 30 11:18:29 EST 2011<br />
setenv GMS_PATH /home/scinet/gpc/src/gamess-Aug1811R1<br />
# machine type<br />
setenv GMS_TARGET linux64<br />
# FORTRAN compiler setup<br />
setenv GMS_FORTRAN ifort<br />
setenv GMS_IFORT_VERNO 12<br />
# mathematical library setup<br />
setenv GMS_MATHLIB mkl<br />
setenv GMS_MATHLIB_PATH /scinet/gpc/intel/ics/composer_xe_2011_sp1.6.233/mkl/lib/intel64<br />
setenv GMS_MKL_VERNO "12-so"<br />
# parallel message passing model setup<br />
setenv GMS_DDI_COMM mpi<br />
setenv GMS_MPI_LIB impi<br />
setenv GMS_MPI_PATH /scinet/gpc/intel/impi/4.0.2.003<br />
<br />
<br />
The code was built with the "compall" script, and then linked using the "lked" script. A new module, gamess/Aug1811R1, was created for CentOS6:<br />
<br />
[dgruner@gpc-f103n058 gamess-Aug1811R1]$ module show gamess<br />
-------------------------------------------------------------------<br />
/scinet/gpc/Modules6/Modules/modulefiles/gamess/Aug1811R1:<br />
<br />
module-whatis adds GAMESS (US), Aug 18/11 R1, environment variables <br />
prereq intel <br />
prereq intelmpi <br />
setenv SCINET_GAMESS_HOME /scinet/gpc/Applications/gamess-Aug1811R1 <br />
setenv SCINET_RUNGMS /scinet/gpc/Applications/gamess-Aug1811R1/rungms <br />
setenv SCINET_GAMESS_SCRATCH /scratch/s/scinet/dgruner/gamess-scratch <br />
-------------------------------------------------------------------<br />
<br />
(note that the $SCINET_GAMESS_SCRATCH directory is defined for each user).<br />
<br />
The $SCINET_RUNGMS run script will work with both ethernet and infiniband connected nodes, so unlike the older version there is no need for specialized scripts depending on the node interconnect. Here are some sample scripts showing how to run gamess on the GPC:<br />
<br />
====Running GAMESS====<br />
- Make sure the directory $SCRATCH/gamess-scratch exists (the $SCINET_RUNGMS script will create it if it does not exist)<br />
<br />
- Make sure the modules: intel, intelmpi, gamess are loaded (in your .bashrc: "module load intel intelmpi gamess").<br />
<br />
- Create a torque script to run GAMESS, as in the examples that follow<br />
<br />
- The GAMESS executable is in $SCINET_GAMESS_HOME/gamess.00.x<br />
- The rungms script is in $SCINET_GAMESS_HOME/rungms (also defined as $SCINET_RUNGMS)<br />
<br />
- The rungms script takes 4 arguments: input file, executable number, number of compute processes, elongations. It always assumes there are 8 processors per node (ppn). Note that you can copy the $SCINET_RUNGMS script to your own directory, and - carefully - modify it to suit your needs (at your own risk!).<br />
<br />
For example, in order to run with the input file /scratch/$USER/gamesstest01, on 8 cpus, and the default version (00)<br />
of the executable on a machine with 8 cores:<br />
<br />
# load the gamess and other required modules in .bashrc<br />
module load intel intelmpi gamess <br />
<br />
# run the program<br />
$SCINET_RUNGMS $SCRATCH/gamesstest01 00 8<br />
<br />
Here is a sample torque script for running a GAMESS calculation, on a single 8-core node. Note the comments regarding the use of the [[User_Ramdisk | ramdisk]], a technique that may speed up calculations significantly. It is restricted to runs on a single node, and there is book-keeping required before the job ends, namely copying any useful output from the [[User_Ramdisk | ramdisk]] (/dev/shm) to the user's directory on the /scratch filesystem.<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#PBS -l nodes=1:ppn=8,walltime=48:00:00,os=centos6computeA<br />
#PBS -N gamessjob<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
# the input file is typically named something like "gamesjob.inp"<br />
# so the script will be run like "$SCINET_RUNGMS gamessjob 00 8"<br />
<br />
# load the gamess module if not in .bashrc already<br />
# module load intel intelmpi gamess<br />
<br />
# If - and only if - the run is on a single node, and does not require too much memory,<br />
# then the runtime scratch directory can be put on ramdisk on the <br />
# compute node. This helps speed up the code, and avoids contention for<br />
# disk resources. Note that the maximum size of the ramdisk is 8 GB,<br />
# and the maximum total memory available to jobs is about 14 GB per node<br />
# (including program memory and ramdisk).<br />
#<br />
# In order to use the ramdisk uncomment the following definition:<br />
#export SCINET_GAMESS_SCRATCH=/dev/shm<br />
<br />
# run the program<br />
<br />
$SCINET_RUNGMS gamessjob 00 8<br />
<br />
# copy any files of interest from the $SCINET_GAMESS_SCRATCH directory<br />
# to a "real" directory on disk, e.g. /scratch/$USER/gamessjob<br />
# cp $SCINET_GAMESS_SCRATCH/* /scratch/$USER/gamessjob<br />
</source><br />
<br />
<br />
Here is a similar script, but using 2 nodes with infiniband interconnect:<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#PBS -l nodes=2:ib:ppn=8,walltime=48:00:00,os=centos6computeA<br />
#PBS -N gamessjob<br />
<br />
## To submit type: qsub gmsib.sh<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
# the input file is typically named something like "gamesjob.inp"<br />
# so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"<br />
<br />
# load the gamess module if not in .bashrc alreadyc<br />
module load intel intelmpi gamess<br />
<br />
# This script requests InfiniBand-connected nodes (:ib above). The IntelMPI should run with the best transport it can find (InfiniBand), and will default back to ethernet if running on ethernet-only nodes.<br />
<br />
$SCINET_RUNGMS gamessjob 00 16<br />
</source><br />
<br />
<br />
<br />
<br />
====Notes on older version, May 22, 2009, built on CentOS5====<br />
<br />
<br />
The GAMESS version January 12, 2009 R3 was built using the Intel v11.1 compilers and v3.2.2 MPI library, according to the instructions in http://software.intel.com/en-us/articles/building-gamess-with-intel-compilers-intel-mkl-and-intel-mpi-on-linux/<br />
<br />
The required build scripts - comp, compall, lked - and run script - rungms - were modified to account for our own installation. In order to build GAMESS one first must ensure that the intel and intelmpi modules are loaded ("module load intel intelmpi"). This applies to running GAMESS as well.<br />
The module "gamess" must also be loaded in order to run GAMESS ("module load gamess").<br />
<br />
The modified scripts are in the file /scinet/gpc/src/gamess-on-scinet.tar.gz<br />
<br />
====Running GAMESS====<br />
- Make sure the directory /scratch/$USER/gamess-scratch exists (the $SCINET_RUNGMS script will create it if it does not exist)<br />
<br />
- Make sure the modules: intel, intelmpi, gamess are loaded (in your .bashrc: "module load intel intelmpi gamess").<br />
<br />
- Create a torque script to run GAMESS. Here is an example:<br />
<br />
- The GAMESS executable is in $SCINET_GAMESS_HOME/gamess.00.x<br />
- The rungms script is in $SCINET_GAMESS_HOME/rungms (actually it is $SCINET_RUNGMS)<br />
<br />
- For multinode runs, use the $SCINET_RUNGMS_IB script to run on the InfiniBand portion of the GPC<br />
<br />
- The rungms script takes 4 arguments: input file, executable number, number of compute processes, processors per node<br />
<br />
For example, in order to run with the input file /scratch/$USER/gamesstest01, on 8 cpus, and the default version (00)<br />
of the executable on a machine with 8 cores:<br />
<br />
# load the gamess and other required modules in .bashrc<br />
module load gcc intel intelmpi gamess <br />
<br />
# run the program<br />
$SCINET_RUNGMS /scratch/$USER/gamesstest01 00 8 8<br />
<br />
Here is a sample torque script for running a GAMESS calculation, on a single 8-core node. Note the comments regarding the use of the [[User_Ramdisk | ramdisk]], a technique that may speed up calculations significantly. It is restricted to runs on a single node, and there is book-keeping required before the job ends, namely copying any useful output from the [[User_Ramdisk | ramdisk]] (/dev/shm) to the user's directory on the /scratch filesystem.<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#PBS -l nodes=1:ppn=8,walltime=48:00:00,os=centos53computeA<br />
#PBS -N gamessjob<br />
<br />
## To submit type: qsub gms.sh<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
# the input file is typically named something like "gamesjob.inp"<br />
# so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"<br />
<br />
# load the gamess module if not in .bashrc already<br />
# module load gcc intel intelmpi gamess<br />
<br />
# If - and only if - the run is on a single node, and does not require too much memory,<br />
# then the runtime scratch directory can be put on ramdisk on the <br />
# compute node. This helps speed up the code, and avoids contention for<br />
# disk resources. Note that the maximum size of the ramdisk is 8 GB,<br />
# and the maximum total memory available to jobs is about 14 GB per node<br />
# (including program memory and ramdisk).<br />
#<br />
# In order to use the ramdisk uncomment the following definition:<br />
#export SCINET_GAMESS_SCRATCH=/dev/shm<br />
<br />
# run the program<br />
<br />
$SCINET_RUNGMS gamessjob 00 8 8<br />
<br />
# copy any files of interest from the $SCINET_GAMESS_SCRATCH directory<br />
# to a "real" directory on disk, e.g. /scratch/$USER/gamessjob<br />
# cp $SCINET_GAMESS_SCRATCH/* /scratch/$USER/gamessjob<br />
</source><br />
<br />
Here is a similar script, but this one uses 2 InfiniBand-connected nodes,<br />
and runs the appropriate $SCINET_RUNGMS_IB script to actually run the job:<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#PBS -l nodes=2:ib:ppn=8,walltime=48:00:00,os=centos53computeA<br />
#PBS -N gamessjob<br />
<br />
## To submit type: qsub gmsib.sh<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
# the input file is typically named something like "gamesjob.inp"<br />
# so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"<br />
<br />
# load the gamess module if not in .bashrc already<br />
# actually, it MUST be in .bashrc<br />
# module load gamess<br />
<br />
# This script requests InfiniBand-connected nodes (:ib above)<br />
# so it must run with the IB version of the rungms script,<br />
# $SCINET_RUNGMS_IB<br />
<br />
$SCINET_RUNGMS_IB gamessjob 00 16 8<br />
</source><br />
<br />
<br />
-- [[User:Dgruner|dgruner]] 5 October 2009<br />
-- [[User:Dgruner|dgruner]] Updated on 14 September 2010</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=SciNet_Usage_Reports&diff=9418SciNet Usage Reports2018-08-31T23:40:44Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
SciNet usage reports are now available on the CCDB website - <br />
https://ccdb.computecanada.ca/. Login and then click on "View Group Usage"<br />
under the "My Account" menu.<br />
Please let us know if you have comments/questions about these reports.<br />
<br />
PIs and all people that they sponsor will be able to view the reports <br />
for only their group.<br />
<br />
There is a summary report which gives a high-level <br />
overview for those groups with RAC allocations as well as more <br />
detailed cumulative reports which breakdown usage by allocation, system <br />
and user. These are all updated nightly.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=HPSS_compared_to_HSM-TSM&diff=9417HPSS compared to HSM-TSM2018-08-31T23:39:53Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
'''Understanding HPSS by comparing it with HSM-TSM at SciNet'''<br />
<br />
The [[HSM#Basic_Concepts|Basic Concepts]] and [[HSM#Definitions|Definitions]] described for HSM-TSM are equally applicable to HPSS.<br />
<br />
For most part both systems are very similar in topology and functionality. They both offer a way to offoad/archive data from the most active file systems (scratch and project) without necessarily having to deal directly with the tape library or "tape commands". The main differences are in the [https://support.scinet.utoronto.ca/wiki/index.php/HPSS_terminology terminology] and the [https://support.scinet.utoronto.ca/wiki/index.php?title=HPSS_Servers&action=edit&redlink=1 servers scalability] of HPSS.<br />
<br />
== machines used in the setup ==<br />
* HSM-TSM at SciNet requires 2 nodes: 1 tapenode (which also runs TSM and HSM servers), 1 datamover (running the HSM client). This setup is not easily scalable, and performance is limited.<br />
* HPSS requires a minimum of 4 nodes: 1 core, 1 mover (we currently have 2), 1 HSI/HTAR server and 1 gateway (archive01), which functions more like a datamover under HSM, ie, the node users can login in order to transfer files. The setup is designed to be scalable, just by adding more movers, gateways, cache LUNs to the SAN, larger tape library, more tape drives, larger/more DB volumes. Currently, the HSI/HTAR server and the core are configured on the same node.<br />
<br />
== Both have a component of disk cache and another of tapes (inaccessible to end-users): ==<br />
* the cache under HSM-TSM is 15TB, formated as gpfs, and mounted as /repository on datamover2<br />
* the cache under HPSS is 233TB, SAN attached to the 2 movers, proprietary format, but it can be accessed as /archive through the HSI prompt on gpc-archive01, provided that you submit your queries via the queue system<br />
<br />
== migration between cache and tapes: ==<br />
* under HSM-TSM, tapenode is the system directly connected to a [http://www-03.ibm.com/systems/storage/tape/ts3310/ TS3310 tape library], it has the function of moving data back and forth between cache (/repository) and tapes. This migration can be done automatically (by the HSM daemons running on datamover2) or manually from a shell on datamover2 by the users, with dsmmigrate or dsmrecall commands. <br />
* under HPSS the tapenode equivalent is known as mover, and we have 2 of them. They are the ones connected to the [http://www-03.ibm.com/systems/storage/tape/ts3500/ TS3500 tape library]. For most part there is no need for user direct interference on this migration, which is done automatically between the cache and the tapes, and coordinated by the core node. However the user has the option to manually migrate and recall files and directories. The term migrate is used predominantly in the context of copying files down the hierarchy to tapes. After migration, files may be deleted from the cache via purge operations. Files are copied up the hierarchy via staging operations, usually when accessed after being purged from the top level of the hierarchy (HSI 'migrate', 'stage' and 'purge'). At SciNet we have all cache<=>tapes transfers set to be done automatically<br />
<br />
== relocation of data from /scratch and /project to the cache. Access to the cache by users. ==<br />
* HSM-TSM has a system called datamover2 specially for this relocation purpose. Users can login to it using ssh, and rely on unix commands such as cp, mv, rsync or tar to transfer data from /scratch and /project to the cache (/repository)<br />
* under HPSS all access to the system is done through the [[Moab|GPC queue system]]. Job scripts interact with the cache through a node known as the gateway machine, named gpc-archive01. All tools available in HSM-TSM for transferring files to/from repository on datamover2 (cp, mv, tar, rsync, rcp, sftp, gridFTP, etc) have [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help an equivalent] on gpc-archive01 through the HSI interface. In addition there is also a tar equivalent application called HTAR.<br />
<br />
== exporting and accessing the cache ==<br />
* under HSM-TSM the cache is a gpfs, and in theory it could be mounted across all 4000 nodes in the cluster. In practice we only mount it on 2 nodes, datamover2 and tapenode as /repository. In this scenario, although possible, there is no real need for scp, sftp or rsync remotely. Users can access the cache directly with a ssh session to datamover2.<br />
* under HPSS the cache is not a parallel file system. It's a proprietary file system (hpssfs) and can only be accessed in its native raw format on the 2 movers, the core and the hsi server systems. On the gateway machine we could mount it locally in the traditional fashion as a virtual file system (VFS), but this capability is currently disabled at SciNet. However the HSI prompt on gpc-archive01 allows the access of the naming space on the cache as /archive.<br />
<br />
== Metadata and database: ==<br />
* In TSM-HSM there exists a DB2 to keep track of all material on tapes and it runs on tapenode. The metadata for file/stubs on cache is kept within the cache gpfs itself on a .SpaceMan? directory, and it's maintained by datamover2.<br />
* HPSS has a set of drives (2x 144TB LUNs mirrored) directly attached to the core node, just to store all information related to naming space an ownership inside the cache. The core node also runs a DB2 holding information on all material on tapes as well as info on the metadata.<br />
<br />
== Configuration and logistics definitions ==<br />
* TSM-HSM uses a set of policies to determine which files/stub should be migrated, and when. Files can be grouped in tape pools primarily based on their paths on the file systems, but it's nearly impossible to segregate contents in the tape-pools based on the individual ownership/groups of the files.<br />
* The HPSS equivalent to policies is a combination of Classes of Services (COS) and Storage Class, and the performance of migrations between the original location of the files, through the cache and into tapes is very dependent on how the cache is physically stripped, the configuration parameters from the Storage Classes as well as the number of small files involved. The equivalent to tape-pools in HPSS is known as Families. There is also provision to have files migrated to a "mirrored tape" COS, which is the default at SciNet.<br />
<br />
== Handling small file in the cache ==<br />
* Under TSM-HSM, it's impossible to prevent or enforce the non-placement of small files in cache (/repository is gpfs). All we can do is rely on user's education and outreach, so that they won't transfer all the small files they have on the active file systems (scratch and project) to /repository. As a more efficient approach we make recommendations and encourage users to generate tar-balls larger than 10GB, and prevent the proliferation of small files inside the cache that way.<br />
* Under the proprietary HPSS file system we face the same issue of not being able to prevent or enforce the non-placement of small files in cache. One effective workaround utility is HTAR, used for aggregating a set of files from the local file system directly into the HPSS cache, creating a file that conforms to the POSIX TAR specification. When HTAR creates the TAR file, it also builds an index file ".idx", which is stored in the cache along with the associated TAR file. HPSS also supports its own small file tape aggregation. By aggregating many, many small files in the cache into a single object, tape drives are kept moving for longer periods of time (streaming) -- greatly improving tape drive performance!<br />
<br />
== files and stubs in cache ==<br />
* Under TSM-HSM, when files are migrated to tapes and the original files are purged from the cache, what is left behind is the directory tree with the HSM stub files and the metadata associated with them. It's possible to distinguish between resident and purged files by using the 'dsmls' command.<br />
* Under HPSS when a file or directory is migrated and subsequently purged the cache stays empty, unlike HSM, where a zero-byte stub remains. In fact, this would be the main difference between TSM-HSM and HPSS. The HPSS Core Server maintains the HPSS name space in system metadata. However, from the HSI prompt we can still query the metadata/DB2 server indirectly with standard unix command such as ls, pwd, du, etc, and get a response from the system as if the files were really there.<br />
<br />
<br />
[[Data Management|BACK TO Data Management]]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Cpmd&diff=9416Cpmd2018-08-31T23:39:36Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
====CPMD====<br />
<br />
<br />
The CPMD version 3.13.2 package was built using the Intel v11.1 compilers, and the OpenMPI v1.4.1 library.<br />
<br />
Patched with the latest available patch. In the SOURCE directory:<br />
<br />
patch -p2 < ../../cpmd-3.13.2_01.patch<br />
<br />
<br />
Basic configuration used:<br />
<br />
<source lang="bash"><br />
#INFO#<br />
#INFO# Configuration to build a parallel cpmd executable for a linux machine<br />
#INFO# with an AMD64/EM64T cpu (Opteron/AthlonFX/Athlon64/Xeon-EM64T) using <br />
#INFO# the Intel Fortran Compiler with EM64T extensions.<br />
#INFO#<br />
#INFO# For optimal performance you should use a specifically tuned BLAS/LAPACK<br />
#INFO# library. This example uses the Intel MKL library.<br />
#INFO#<br />
#INFO# see http://www.theochem.ruhr-uni-bochum.de/~axel.kohlmeyer/cpmd-linux.html<br />
#INFO# for more information on compilind and running CPMD on linux machines.<br />
#INFO#<br />
#INFO# NOTE: CPMD cannot be compiled with the GNU Fortran compiler.<br />
#INFO#<br />
IRAT=2<br />
CFLAGS='-O2 -Wall -m64'<br />
CPP='/lib/cpp -P -C -traditional'<br />
CPPFLAGS='-D__Linux -D__PGI -DFFT_DEFAULT -DPOINTER8 -DLINUX_IFC \<br />
-DPARALLEL -DMYRINET'<br />
FFLAGS='-pc64 -O2 -unroll'<br />
LFLAGS=' -L. -L${MKLPATH} ${MKLPATH}/libmkl_solver_lp64_sequential.a -Wl,--start-group \<br />
-lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -lpthread'<br />
FFLAGS_GROMOS='-Dgood_luck $(FFLAGS)'<br />
if [ $debug ]; then<br />
FC='mpif77 -c -g'<br />
CC='mpicc -g -Wall -m64'<br />
LD='mpif77 -g'<br />
else<br />
FC='mpif77 -c '<br />
CC='mpicc'<br />
LD='mpif77 -static-intel '<br />
fi<br />
</source><br />
<br />
<br />
In the SOURCE directory do:<br />
<br />
./config.sh LINUX_INTEL64_INTEL_MPI > Makefile<br />
Make >& make.out<br />
<br />
Created cpmd/3.13.2 module:<br />
<br />
<source lang="tcl"><br />
#%Module -*- tcl -*-<br />
<br />
# CPMD 3.13.2<br />
<br />
proc ModulesHelp { } {<br />
puts stderr "\tThis module adds CPMD environment variables"<br />
}<br />
<br />
module-whatis "adds CPMD environment variables"<br />
<br />
# CPMD was compiled with Intel compilers and OpenMPI<br />
prereq intel <br />
prereq openmpi<br />
<br />
setenv SCINET_CPMD_HOME /scinet/gpc/Applications/cpmd/3.13.2<br />
setenv SCINET_CPMD_BIN /scinet/gpc/Applications/cpmd/3.13.2/bin<br />
append-path PATH /scinet/gpc/Applications/cpmd/3.13.2/bin<br />
setenv CPMD_PP_LIBRARY_PATH /scinet/gpc/Applications/cpmd/3.13.2/PPLIBNEW<br />
</source><br />
<br />
<br />
====Running CPMD====<br />
<br />
- Load the necessary modules: gcc, intel, intelmpi, nwchem (best done in your .bashrc)<br />
<br />
# module load intel openmpi cpmd<br />
<br />
- The CPMD executable is in $SCINET_CPMD_BIN/cpmd.x<br />
<br />
- For multinode runs, use the sample IB script futher below<br />
<br />
- Create a torque script to run CPMD. Here is an example for a calculation on a single 8-core node:<br />
<br />
- The standard CPMD pseudopotentials are in the directory $CPMD_PP_LIBRARY_PATH. The following sample scripts use a user-defined pseudopotential.<br />
<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#MOAB/Torque submission script for Scinet GPC (ethernet)<br />
#PBS -l nodes=1:ppn=8,walltime=00:30:00<br />
#PBS -N cpmdjob<br />
# load the cpmd and other required modules if not in .bashrc already<br />
module load intel openmpi cpmd<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
mpirun -np 8 -hostfile $PBS_NODEFILE $SCINET_CPMD_BIN/cpmd.x inp1 /home/mgalib/uspp/uspp-736/Pot >out1<br />
</source><br />
<br />
Here is a similar script, but this one uses 2 InfiniBand-connected nodes:<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#PBS -l nodes=2:ib:ppn=8,walltime=48:00:00<br />
#PBS -N cpmdjob<br />
<br />
## To submit type: qsub nwc.sh (where nwc.sh is the name of this script)<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
mpirun -np 16 -hostfile $PBS_NODEFILE $SCINET_CPMD_BIN/cpmd.x inp1 /home/mgalib/uspp/uspp-736/Pot >out1<br />
</source><br />
<br />
<br />
-- [[User:Dgruner|dgruner]] 3 September 2010</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Performance_And_Debugging_Tools:_TCS&diff=9415Performance And Debugging Tools: TCS2018-08-31T23:38:37Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==Memory Profiling ==<br />
<br />
===PurifyPlus===<br />
This is IBM/Rational's set of tools for memory profiling and code testing. Specifically, the tools are: purify, purecov and quantify.<br />
<br />
In order to gain access to these tools, you need to source the script that defines them to your shell:<br />
<br />
<tt> source /scinet/tcs/Rational/purifyplus_setup.sh </tt><br />
<br />
The documentation for getting started with these tools is in the directory:<br />
<br />
<tt> /scinet/tcs/Rational/releases/PurifyPlus.7.0.1/docs/pdf </tt><br />
<br />
or here: [http://support.scinet.utoronto.ca/Manuals/PurifyPlusGettingStarted_en.pdf PurifyPlus Getting Started].<br />
<br />
==Debugging==<br />
<br />
===dbx===<br />
<br />
[http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/dbx_symbolic_debug.htm dbx] is source-language level serial debugger for aix; it works more or less like gdb for stepping through code. For it to be useful, you have to have compiled with symbol information in the executable, eg with <tt>-g</tt>.<br />
<br />
==Performance Profiling==<br />
<br />
===gprof===<br />
<br />
[http://sourceware.org/binutils/docs-2.19/gprof/index.html gprof] is a very useful tool for finding out where a program is spending its time; its use is described in our [[Introduction_To_Performance#gprof_.28profiling:_everywhere.29 | Intro To Performance]]. You compile with <tt>-g -pg</tt> and run the program as usual; a file named <tt>gmon.out</tt> is created, which one can use via the <tt>gprof</tt> program to see a profile of your program's execution time.<br />
<br />
===Xprofiler===<br />
<br />
<tt>Xprof</tt> is an AIX utility for graphically displaying the output of profiling information; it's equivalent to gprof but gives a better `bird's-eye view' of large and complex programs. You compile your program with <tt>-pg -g</tt> as before but run <tt>Xprof</tt> on the output:<br />
<pre><br />
Xprof program_name gmon.out<br />
</pre><br />
<br />
==Performance counters==<br />
<br />
===hpmcount===<br />
<br />
On the TCS, hpmcount allows the querying of the performance counter values in the CPUs themselves over the course of a run. Since here we are simply asking the CPU to report values it obtains during the run of a program, the code does not need to be instrumented; simply typing<br />
<pre><br />
hpmcount hpmcount_args program_name program_args<br />
</pre><br />
runs the job as normal and reports the counter results at the end of the run. <br />
<pre><br />
hpmcount -d -s 5,12 program_name program_args<br />
</pre> <br />
reports counter sets 5 and 12, which are very useful for showing memory performance (showing L1 and L2 cache misses) and set 6 is especially useful for shared memory profiling, giving statistics about how often off-processor memory had to be accessed. More details are available on our [[ Introduction_To_Performance#hpmcount_.28performance_counters:_TCS.29 | Introduction to Performance ]] or from the [http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.cmds/doc/aixcmds2/hpmcount.htm IBM website].<br />
<br />
<br />
==Performance logging==<br />
<br />
===PeekPerf===<br />
<br />
[[Image:PeekPerf.png|thumb|An example of using PeekPerf]]<br />
<br />
[http://domino.research.ibm.com/comm/research_projects.nsf/pages/hpct.index.html Peekperf] is IBM's single graphical `dashboard' providing access to many performance measurement tools for exmaining Hardware Counter data, threads, message passing, IO, and memory access, several of which are available seperately as command-line tools. Its use is described in our [[Introduction_To_Performance#PeekPerf_.28profiling.2C_TCS.29 | Intro to Performance]].<br />
<br />
===MPE/Jumpshot===<br />
<br />
MPI defines a profiling layer which can be used to intercept MPI calls and log information about them. This is used by the peekperf package above to instrument code by inserting function calls. The same library and tools can be used manually: eg<br />
<pre><br />
$ mpCC -pg a.c -o a.out -L/usr/lpp/ppe.hpct/lib64 -lmpitrace<br />
$ mpiexec -n 2 ./a.out-hostfile HOSTFILE<br />
$ peekview result.viz<br />
</pre><br />
(where HOSTFILE is a file containing the TCS devel node hostname; see [[FAQ#MPI_development_and_interactive_testing | our FAQ entry for interactive running on the TCS nodes]].)<br />
The above will use peekview to show the results for up to three of the MPI tasks (those with the minimum, median, and maximum MPI time, respectively); to get more output, one can set the environment variable<br />
<pre><br />
export OUTPUT_ALL_RANKS=yes<br />
</pre><br />
More information is available [https://wiki.alcf.anl.gov/index.php/HPCT_MPITRACE online].<br />
<br />
[[Image:Jumpshotoutput.png|thumb|jumpshot, part of the MPE development package, can give graphical overviews of the MPI behaviour of programs]]<br />
<br />
Another tool which performs the same task but generates more detailed output is the [http://www.mcs.anl.gov/research/projects/perfvis/ MPE tools] put out by Argonne National Lab. On the TCS, these can be accessed by typing<br />
<pre><br />
module load mpe<br />
</pre><br />
<br />
and then using the mpe wrappers to compile your program: eg<br />
<pre><br />
module load mpe<br />
mpefc -o a.out a.F -mpilog <br />
mpecc -o a.out a.c -mpilog <br />
</pre><br />
Running your program as above will generate a log file named something like <tt>Unknown.clog2</tt>; you can convert this to a format suitable for viewing and then use the <tt>jumpshot</tt> interactive trace viewing tool,<br />
<pre><br />
clog2TOslog2 Unknown.clog2<br />
jumpshot Unknown.slog2<br />
</pre><br />
Note that this tries to open xwindows, so you will have had to login with <tt>ssh -X</tt> or <tt>ssh -Y</tt> into both <tt>login.scinet</tt> and <tt>tcs01</tt> or <tt>tcs02</tt>.<br />
<br />
To use MPE logging on a batch job, you must specify the MPE path and library path in your batch script:<br />
<br />
<pre><br />
#<br />
# LoadLeveler submission script for SciNet TCS: MPI job<br />
#<br />
#@ job_name = testmpe<br />
#@ initialdir = /scratch/YOUR/DIRECTORY<br />
#<br />
#@ tasks_per_node = 64<br />
#@ node = 2<br />
#@ wall_clock_limit= 0:10:00<br />
#@ output = $(job_name).$(jobid).out<br />
#@ error = $(job_name).$(jobid).err<br />
#<br />
#@ notification = complete<br />
#@ notify_user = user@example.com<br />
#<br />
# Don't change anything below here unless you know exactly<br />
# why you are changing it.<br />
#<br />
#@ job_type = parallel<br />
#@ class = verylong<br />
#@ node_usage = not_shared<br />
#@ rset = rset_mcm_affinity<br />
#@ mcm_affinity_options = mcm_distribute mcm_mem_req mcm_sni_none<br />
#@ cpus_per_core=2<br />
#@ task_affinity=cpu(1)<br />
#@ environment = COPY_ALL; MEMORY_AFFINITY=MCM; MP_SYNC_QP=YES; \<br />
# MP_RFIFO_SIZE=16777216; MP_SHM_ATTACH_THRESH=500000; \<br />
# MP_EUIDEVELOP=min; MP_USE_BULK_XFER=yes; \<br />
# MP_RDMA_MTU=4K; MP_BULK_MIN_MSG_SIZE=64k; MP_RC_MAX_QP=8192; \<br />
# PSALLOC=early; NODISCLAIM=true<br />
#<br />
<br />
MPEDIR=/scinet/tcs/mpi/mpe2-1.0.6<br />
export PATH=${MPEDIR}/bin:${MPEDIR}/sbin:${PATH}<br />
export LD_LIBRARY_PATH=${MPEDIR}/lib:${LD_LIBRARY_PATH}<br />
<br />
YOUR-PROGRAM<br />
<br />
#@ queue<br />
</pre><br />
<br />
===Scalasca===<br />
<br />
[http://www.fz-juelich.de/jsc/scalasca/ Scalasca] is a sophisticated tool for analyzing performance and finding common performance problems, installed on both TCS and GPC. We describe it in our [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Intro to Performance]].</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=SciNet_User_Support&diff=9414SciNet User Support2018-08-31T23:38:13Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
=== Support team ===<br />
SciNet's [http://www.scinethpc.ca/staff support team of system administrators and technical analysts] can help you with wide range problems such as<br />
* how to use SciNet's systems,<br />
* how to set up your runs most efficiently,<br />
* how to parallize or optimize your code,<br />
* using and installing libraries.<br />
SciNet support staff is located at [http://www.scinethpc.ca/2010/08/contact-us/ 256 McCaul St] on the St. George Campus of the University of Toronto.<br />
<br />
=== Courses ===<br />
SciNet organizes a number of types of classes for its users:<br />
* Intro to SciNet (1 hour)<br />
* SciNet User Group meetings including 30 minute TechTalks, every second Wednesday on the month at noon.<br />
* Full day courses on e.g. parallel I/O, (parallel) programming, etc.<br />
* 5 day course on parallel programming (openMP, MPI)<br />
Read more on these on the [https://support.scinet.utoronto.ca/education Training and Education web site], where you can also register ("enroll").<br />
<br />
Many past lecture slide and some videos can be found on the [[Knowledge Base: Tutorials and Manuals]] page.<br />
<br />
=== How to request help ===<br />
If you have problems, questions, or requests, and you couldn't find the answer in this wiki, send an e-mail '''with all relevant information''' to:<br />
<br />
:[mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] <br />
<br />
'''''For connection problems, include:'''''<br />
* Where you are connecting from,<br />
* Where you are trying to connect to<br />
* The type of computer you are connecting from<br />
<br />
'''''For failed jobs, include:'''''<br />
* What cluster you were using<br />
* A description of what the run is supposed to do<br />
* Job ID<br />
* Error messages you got<br />
* Full path to the job script<br />
* Full path to your code<br />
* Location of input/error files (please do not delete these!)<br />
<br />
'''''For failed compilations, include:'''''<br />
* What cluster you were using<br />
* Full path to the code and Makefiles<br />
* A description of what the code is supposed to do<br />
* Error messages you got<br />
<br />
'''''To make an appointment with one of our analysts:'''''<br />
* Use the same e-mail, [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] <br />
* There is no formal ticketing system.<br />
*Avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team.<br />
<br />
'''Note that your password should never, never, never be to sent to us, even if your question is about your account.'''</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=MARS&diff=9413MARS2018-08-31T23:37:37Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
MARS is not more. [https://support.scinet.utoronto.ca/wiki/index.php/HPSS Follow this link]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=HPSS_Servers&diff=9412HPSS Servers2018-08-31T23:36:46Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
HPSS servers include the Core Server, Migration/Purge Server, Gatekeeper, Location Server, Log Client, Log Daemon, Physical Volume Library, Physical Volume Repository, Mover, Storage System Management System Manager, and Startup Daemon. Figure 3: HPSS Components provides a simplified view of the HPSS system. Each major server component is shown, along with the basic control communication paths (thin arrowed lines). Infrastructure items (those components that “glue together” the distributed servers) are shown at the top of the cube.<br />
<br />
'''''Core Server.''''' The Core Server provides several key sets of functionality.<br />
<br />
First, the Core Server provides translation between human-oriented names and HPSS object identifiers. Name space objects managed by the Core Server are filesets, junctions, directories, files, hard links, and symbolic links. The Core Server provides access verification to objects and mechanisms for manipulating access to these objects via a Portable Operating System Interface (POSIX) view of the name space. This name space is a hierarchical structure consisting of directories, files, and links. These name space objects may exist within filesets that are connected via junctions.<br />
<br />
Second, the Core Server provides the abstraction of logical bitfiles to its clients. A bitfile is identified by a Core Server generated name called a bitfile ID. Clients may reference portions of a bitfile by specifying the bitfile ID and a starting address and length. The Core Server supports random access to files and sparsely written files. It supports parallel reading and writing of data to bitfiles and performs the mapping of logical portions of bitfiles onto physical storage devices. The Core Server supports the migration, purging, and staging of data in a storage hierarchy (though the migration/purge policies are implemented through the Migration/Purge Server, a client to the Core Server).<br />
<br />
Third, the Core Server provides a hierarchy of storage objects: storage segments, virtual volumes, and physical volumes. The Core Server translates storage segment references into virtual volume references and then into physical volume references, handles the mapping of physical resources into striped virtual volumes to allow parallel I/O to that set of resources, and schedules the mounting and dismounting of removable media through the Physical Volume Library (see below).<br />
<br />
'''''Migration/Purge Server (MPS).''''' The MPS allows a site to implement its storage management policies by managing the placement of data on HPSS storage media using site-defined migration and purge policies. By making appropriate calls to its Core Server, an MPS copies data to lower levels in the hierarchy (migration), removes data from the current level once copies have been made (purge), or moves data between volumes at the same level (lateral move). Based on the hierarchy configuration, MPS can be directed to create duplicate copies of data when it is being migrated from disk or tape. This is done by copying the data to multiple lower levels in the storage hierarchy.<br />
<br />
There are three types of migration: disk migration, tape file migration, and tape volume migration. The designation disk or tape refers to the type of storage class that migration is running against. See Section 3.7.2: Migration/Purge Server on page 72 for a more complete discussion of the different types of migration.<br />
<br />
MPS runs migration on each storage class periodically using the time interval specified in the migration policy for that class. See Section 2.3.7: HPSS Policy Modules on page 34 for details on migration and purge policies. Migration runs can be started automatically when the warning or critical space thresholds for the storage class are exceeded. In addition, migration runs can be started manually by an administrator.<br />
<br />
Purge runs are started automatically on each storage class when the free space in that class falls below the percentage specified in the purge policy. Purge runs may also be started manually.<br />
<br />
'''''Disk Migration/Purge:'''''<br />
The purpose of disk migration is to make one or more copies of disk files to lower levels in the hierarchy. The number of copies depends on the configuration of the hierarchy. For disk, migration and purge are separate operations. It is common for disk storage classes which have been configured for migration to also be configured for purge as well. Once a file has been migrated (copied) downwards in the hierarchy, it becomes eligible for purge, which subsequently removes the file from the higher level and allows the disk space to be reused.<br />
<br />
'''''Tape File Migration:'''''<br />
The purpose of tape file migration is to make an additional copy (or multiple additional copies) of a file, in a tape storage class, to a lower level in the hierarchy. It is also possible to move files downwards instead of copying them. In this case there is no duplicate copy maintained. There is no separate purge component to tape file migration. Empty volumes must be reclaimed using the reclaim utility.<br />
<br />
'''''Tape Volume Migration:'''''<br />
The purpose of tape volume migration is to free tape volumes for reuse. Tape volumes are selected based on being in the EOM map state and containing the most unused space (caused by users overwriting or deleting files). The remaining segments on these volumes are either migrated downwards to the next level in the hierarchy, or are moved laterally to another tape volume at the same level. This results in empty tape volumes which may then be reclaimed. Note that there is no purge component to tape volume migration. All of the operations use a move instead of a copy semantic.<br />
<br />
'''''Gatekeeper (GK).''''' The Gatekeeper provides two main services:<br />
* It provides sites with the ability to schedule the use of HPSS resources using the Gatekeeping Service.<br />
* It provides sites with the ability to validate user accounts using the Account Validation Service.<br />
<br />
Both of these services allow sites to implement their own policy.<br />
<br />
The default Gatekeeping Service policy is to not do any gatekeeping. Sites may choose to implement a policy for monitoring authorized callers, creates, opens, and stages. The Core Server will call the appropriate GK API depending on the requests that the site-implemented policy is monitoring.<br />
<br />
The Account Validation Service performs authorizations of user storage charges. A site may perform no authorization, default authorization, or site-customized authorization depending on how the Accounting Policy is set up and whether or not a site has written site-specific account validation code. Clients call this service when creating files, changing file ownership, or changing accounting information. If Account Validation is enabled, the Account Validation Service determines if the user is allowed to use a specific account or gives the user an account to use, if needed. The Core Server also calls this service to perform an authorization check just before account-sensitive operations take place.<br />
<br />
'''''Location Server (LS).''''' The Location Server acts as an information clearinghouse to its clients through the HPSS Client API to enable them to locate servers and gather information from both local and remote HPSS systems. Its primary function is to allow a client to determine a server's location and, by knowing other information about the server such as its object UUID, determine its server type or its subsystem id. This allows a client to contact the appropriate server. Usually the Location Server is only used by the Core Server or the Gatekeeper.<br />
<br />
'''''Physical Volume Library (PVL).''''' The PVL manages all HPSS physical volumes. It is in charge of mounting and dismounting sets of physical volumes, allocating drive and cartridge resources to satisfy mount and dismount requests, providing a mapping of physical volume to cartridge and of cartridge to Physical Volume Repository (PVR), and issuing commands to PVRs to perform physical mount and dismount actions. A primary function of the PVL is the support for atomic mounts of sets of cartridges for parallel access to data. Atomic mounts are implemented by the PVL, which waits until all necessary cartridge resources for a request are available before issuing mount commands to the PVRs.<br />
<br />
'''Physical Volume Repository (PVR).''' PVRs manage HPSS cartridges. Though an HPSS system may contain multiple PVRs, each cartridge is managed by only one. PVRs provide APIs for clients to request cartridge mounts and dismounts and query the status of cartridges. For convenience, PVRs are often configured in one-to-one correspondence to tape libraries.<br />
<br />
For information on the types of tape libraries supported by HPSS PVRs, see Section 3.4.2:Robotically Mounted Tape on page 49.<br />
<br />
An Operator PVR is provided for cartridges not under control of a robotic library. These cartridges are mounted on a set of drives by operators.<br />
<br />
* '''''Mover (MVR).''''' The purpose of the Mover is to transfer data from a source device to a sink device. A device can be a standard I/O device with geometry (e.g., tape, disk) or a device without geometry (e.g., network, memory). The Mover’s client (typically the Core Server) describes the data to be moved and where the data is to be sent. It is the Mover’s responsibility to actually transfer the data, retrying failed requests and attempting to optimize transfers. The Mover supports transfers for disk devices, tape devices and a mover protocol that can be used as a lightweight coordination and flow control mechanism for large transfers.<br />
<br />
* '''''Log Client (LOGC).''''' The Log Client receives log messages from each HPSS server running on its node and filters those messages based on the configured log policies. Log messages which pass the filter are written to a local log and sent on to the Log Daemon.<br />
<br />
* '''''Log Daemon (LOGD).''''' The Log Daemon enters log messages into the central log file and sends those messages which are to be displayed in the Alarms and Events SSM window to the SSM System Manager.<br />
<br />
'''''Storage System Management System Manager (SSMSM).''''' SSM, the Storage System Management subsystem, is the tool used by the system administrator to manage HPSS. SSM has three components, one of which is an HPSS server and two of which are user interface client programs. The server is:<br />
* SSM System Manager (SSMSM, or hpss_ssmsm) - Communicates with all other HPSS components requiring monitoring or control.<br />
<br />
The user interface clients are:<br />
* SSM GUI (hpssgui) - Provides the HPSS administrator or operator the ability to configure or monitor the HPSS System through a graphical user interface.<br />
* SSM Command Line Interface (hpssadm) - Provides the HPSS administrator or operator the ability to configure or monitor a subset of the HPSS system through a set of interactive or batch commands.<br />
* SSM enables the administrator to configure, monitor and control the resources (servers, devices, tape libraries, and media) of HPSS in ways that conform to the management policies of a given customer site.<br />
<br />
Monitoring capabilities include the ability to query the values of important management attributes of storage system resources and the ability to receive notifications of alarms and other significant system events. Controlling capabilities include the ability to start up and shut down servers and the ability to set the values of management attributes of storage system resources and storage system policy parameters. Additionally, SSM can request that specific operations be performed on resources within the storage system, such as adding and deleting logical or physical resources.<br />
<br />
'''''Startup Daemon.''''' The Startup Daemon is called by the SSMSM to start each HPSS server except the SSMSM, the Startup Daemon itself, and the remote portion of the Mover. A Startup Daemon is required on each node where any HPSS Server executes, with the exception that no Startup Daemon is required on nodes where the only HPSS Server is the remote portion of the Mover.<br />
<br />
<br />
[https://support.scinet.utoronto.ca/wiki/index.php/HPSS BACK TO HPSS]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Systems_Overview&diff=9411Systems Overview2018-08-31T23:35:48Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==SciNet Systems==<br />
SciNet is a consortium for High-Performance Computing consisting of researchers at the University of Toronto<br />
and its associated hospitals. It is one of seven such consortia in Canada, which together form [https://computecanada.org/ Compute/Calcul Canada].<br />
<br />
SciNet has two main clusters:<br />
{|-<br />
|<br />
===General Purpose Cluster (GPC)===<br />
[[Image:University_of_Tor_79284gm-a.jpg|left]]<br />
<br />
The [[GPC_Quickstart|General Purpose Cluster]] is an extremely large cluster (ranked 16th in the world at its inception in June 2009, and the fastest in Canada) and is where most simulations are to be done at SciNet. It is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the first in the world to make use of the new chips). The GPC consists of 3,780 nodes with a total of 30,240 2.5GHz cores, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking DDR InfiniBand while the rest of the nodes are connected with 5:1 blocked QDR Infiniband. The compute nodes are accessed through a queuing system that allows jobs with a maximum wall time of 48 hours.<br />
|-<br />
|<br />
<br />
===Tightly Coupled System (TCS)===<br />
<br />
[[Image:TCS-1.jpg|left|160px]]<br />
<br />
The [[TCS_Quickstart|TCS]] is a cluster of `fat' (high-memory, many-core) POWER6 nodes on a very fast Infiniband connection, which was installed in January of 2009. It has a relatively small number of cores (~3000), arranged in nodes of 32 cores with 128 GB of RAM on each node. It is dedicated for jobs that require such a large memory / low latency configuration. Jobs need to use multiples of 32 cores (a node), and are submitted to the LoadLeveler queuing system which allows jobs with a maximum wall time of 48 hours.<br />
|-<br />
|<br />
===Power 7 Linux Cluster (P7)===<br />
<br />
[[Image:IBM755.jpg|right|160px]]<br />
<br />
The [[P7_Linux_Cluster|P7]] is a cluster of 5 (soon to be at least 8) `fat' (high-memory, many-core) POWER7 nodes on a very fast Infiniband connection, which was installed in May of 2011. It has a total of 160 cores, arranged in nodes of 32 cores with 128 GB of RAM on each node. It is dedicated for jobs that require such a large memory / low latency configuration. Jobs need to use multiples of 32 cores or more, and are submitted to the LoadLeveler queuing system which allows jobs with a maximum wall time of 48 hours.<br />
<br />
|-<br />
|<br />
===Accelerator Research Cluster (ARC)===<br />
The [[Accelerator_Research_Cluster|ARC]] is a technology evaluation cluster with a combination of 14 IBM PowerXCell 8i "Cell" nodes and 8 Intel x86_64 "Nehalem" nodes containing 16 NVIDIA M2070 GPUs. The QS22's each have two 3.2GHz "IBM PowerXCell 8i CPU's, where each CPU has 1 Power Processing Unit (PPU) and 8 Synergistic Processing Units (SPU), and 32GB of RAM per node. The Intel nodes have two 2.53GHz 4core Xeon X5550 CPU's with 48GB of RAM per node each containing two NVIDIA M2070 (Fermi) GPU's each with 6GB of RAM.<br />
<br />
Please note that this cluster is not a production cluster and is only accessible to selected users. <br />
<br />
|}</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=9410Namd on BGQ2018-08-31T23:35:02Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (32, 64, 128, etc.). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br />
<br />
The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three "'''Benchmark time'''" lines at the beginning of the simulation's standard output and may not represent long-time averages. The column "cores" in the following results indicates the number of "physical cores" specified by the '''bg_size''' parameter on the submission script.<br />
<br />
== Performance Tuning Benchmarks ==<br />
<br />
Efficiency is measured with respect to the 16 ranks-per-node 32 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|16<br />
|32<br />
|<br />
|2.79<br />
|1.00<br />
|----<br />
|16<br />
|64<br />
|<br />
|5.05<br />
|0.91<br />
|----<br />
|16<br />
|64<br />
|twoAwayX (default)<br />
|5.62<br />
|1.01<br />
|----<br />
|16<br />
|128<br />
|twoAwayX (default)<br />
|10.07<br />
|0.90<br />
|----<br />
|16<br />
|128<br />
|twoAwayXY<br />
|10.59<br />
|0.95<br />
|----<br />
|16<br />
|256<br />
|twoAwayX<br />
|14.32<br />
|0.64<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY (default)<br />
|17.63<br />
|0.79<br />
|----<br />
|16<br />
|256<br />
|twoAwayXYZ<br />
|16.79<br />
|0.75<br />
|----<br />
|16<br />
|512<br />
|twoAwayX<br />
|23.52<br />
|0.53<br />
|----<br />
|16<br />
|512<br />
|twoAwayXY (default)<br />
|25.00<br />
|0.56<br />
|----<br />
|16<br />
|1024<br />
|twoAwayX<br />
|23.67<br />
|0.27<br />
|----<br />
|16<br />
|1024<br />
|twoAwayXY<br />
|28.31<br />
|0.32<br />
|----<br />
|16<br />
|1024<br />
|twoAwayXYZ (default)<br />
|27.98<br />
|0.31<br />
|----<br />
|}<br />
<br />
== PME Pencils ==<br />
A "pencil-based" PME decomposition may be more efficient than the default "slab-based decomposition". In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=8, lblUnload=yes<br />
|12.93<br />
|0.58<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=12, lblUnload=yes<br />
|17.27<br />
|0.77<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=16, lblUnload=yes<br />
|16.02<br />
|0.72<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=20, lblUnload=yes<br />
|15.41<br />
|0.69<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=12<br />
|16.21<br />
|0.73<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=16<br />
|17.92<br />
|0.80<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=20<br />
|17.99<br />
|0.81<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=24<br />
|17.83<br />
|0.80<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=36<br />
|16.97<br />
|0.76<br />
|----<br />
|8<br />
|256<br />
|twoAwayXY, PMEPencils=20<br />
|18.24<br />
|0.82<br />
|----<br />
|16<br />
|256<br />
|twoAwayXY, PMEPencils=20<br />
|17.99<br />
|0.81<br />
|----<br />
|32<br />
|256<br />
|twoAwayXY, PMEPencils=20<br />
|13.94<br />
|0.63<br />
|----<br />
|}<br />
<br />
== Ranks-Per-Node Study ==<br />
The "ranks-per-node" or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations.<br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|32<br />
|32<br />
|<br />
|4.46<br />
|1.6<br />
|----<br />
|32<br />
|64<br />
|<br />
|8.01<br />
|1.43<br />
|----<br />
|32<br />
|128<br />
|<br />
|13.74<br />
|1.3<br />
|----<br />
|32<br />
|256<br />
|<br />
|19.81<br />
|1.12<br />
|----<br />
|}<br />
<br />
== Incorrect Particle-Mesh Ewald Grid ==<br />
<br />
Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the "'''pmeGridSpacing 1.0'''" setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|16<br />
|32<br />
|Poor PME Multiple (144x144x111)<br />
|2.70<br />
|0.97<br />
|----<br />
|16<br />
|64<br />
|Poor PME Multiple (144x144x111)<br />
|5.13<br />
|0.92<br />
|----<br />
|16<br />
|128<br />
|Poor PME Multiple (144x144x111)<br />
|8.61<br />
|0.77<br />
|----<br />
|16<br />
|256<br />
|Poor PME Multiple (144x144x111)<br />
|13.93<br />
|0.62<br />
|----<br />
|16<br />
|512<br />
|Poor PME Multiple (144x144x111)<br />
|17.08<br />
|0.38<br />
|----<br />
|16<br />
|1024<br />
|Poor PME Multiple (144x144x111)<br />
|17.64<br />
|0.20<br />
|----<br />
|}<br />
<br />
== Symmetric multiprocessing Study ==<br />
All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag "+CmiNoProcForComThread" to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br />
<br />
Comparing 32/ppn4 with 64/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 128/ppn4 simulations showed poor scaling so larger systems were not tested.<br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|16<br />
|32<br />
|ppn 1<br />
|2.75<br />
|1.00<br />
|----<br />
|16<br />
|32<br />
|ppn 2<br />
|4.60<br />
|1.67<br />
|----<br />
|16<br />
|32<br />
|ppn 3<br />
|5.59<br />
|2.03<br />
|----<br />
|16<br />
|32<br />
|ppn 4<br />
|6.31<br />
|2.29<br />
|----<br />
|16<br />
|32<br />
|ppn 1, +CmiNoProcForComThread<br />
|2.74<br />
|1.00<br />
|----<br />
|16<br />
|32<br />
|ppn 2, +CmiNoProcForComThread<br />
|4.62<br />
|1.68<br />
|----<br />
|16<br />
|32<br />
|ppn 3, +CmiNoProcForComThread<br />
|5.50<br />
|2.00<br />
|----<br />
|16<br />
|32<br />
|ppn 4, +CmiNoProcForComThread<br />
|6.31<br />
|2.30<br />
|----<br />
|16<br />
|64<br />
|ppn 1<br />
|5.54<br />
|1.01<br />
|----<br />
|16<br />
|64<br />
|ppn 2<br />
|8.31<br />
|1.51<br />
|----<br />
|16<br />
|64<br />
|ppn 3<br />
|8.56<br />
|1.56<br />
|----<br />
|16<br />
|64<br />
|ppn 4<br />
|10.60<br />
|1.92<br />
|----<br />
|16<br />
|64<br />
|ppn 4, +CmiNoProcForComThread<br />
|10.58<br />
|1.92<br />
|----<br />
|16<br />
|128<br />
|ppn 1<br />
|9.93<br />
|0.90<br />
|----<br />
|16<br />
|128<br />
|ppn 2<br />
|13.86<br />
|1.26<br />
|----<br />
|16<br />
|128<br />
|ppn 3<br />
|14.72<br />
|1.34<br />
|----<br />
|16<br />
|128<br />
|ppn 4<br />
|15.10<br />
|1.37<br />
|----<br />
|}<br />
<br />
In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, such as 2,4 or 8, while requesting a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is approximately equal to the performance of the 16-ranks-per-node above. <br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|8<br />
|32<br />
|ppn 1<br />
|1.53<br />
|0.56<br />
|----<br />
|8<br />
|32<br />
|ppn 2<br />
|2.77<br />
|1.01<br />
|----<br />
|8<br />
|32<br />
|ppn 4<br />
|4.64<br />
|1.69<br />
|----<br />
|8<br />
|32<br />
|ppn 8<br />
|6.32<br />
|2.30<br />
|----<br />
|8<br />
|64<br />
|ppn 1<br />
|2.79<br />
|0.51<br />
|----<br />
|8<br />
|64<br />
|ppn 2<br />
|5.59<br />
|1.02<br />
|----<br />
|8<br />
|64<br />
|ppn 4<br />
|8.25<br />
|1.50<br />
|----<br />
|8<br />
|64<br />
|ppn 8<br />
|10.71<br />
|1.95<br />
|----<br />
<br />
|}<br />
<br />
Simulations of both 2 and 4 ranks-per-node offered comparable simulation efficiencies to the other full 64 process jobs in this section. The combination of 32 ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. A simulations of ppn=64 with 1 rank-per-node was not successful, and the ppn=32 job required the use of PMEPencils.<br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|2<br />
|32<br />
|ppn 32<br />
|6.35<br />
|2.29<br />
|----<br />
|2<br />
|32<br />
|ppn 32 (only with PMEPencils 8)<br />
|7.80<br />
|1.95<br />
|----<br />
|4<br />
|32<br />
|ppn 16<br />
|6.30<br />
|2.31<br />
|----<br />
|4<br />
|32<br />
|ppn 16<br />
|10.73<br />
|1.42<br />
|----<br />
|32<br />
|32<br />
|ppn 1, +CmiNoProcForComThread<br />
|4.45<br />
|1.62<br />
|----<br />
|32<br />
|32<br />
|ppn 2, +CmiNoProcForComThread<br />
|6.26<br />
|2.27<br />
|----<br />
|}<br />
<br />
PME pencils offered minimal improvements when selected appropriately. <br />
<br />
{| class="wikitable"<br />
|Ranks<br />
|Cores<br />
|NAMD Config Options<br />
|ns/day<br />
|Efficiency<br />
|----<br />
|16<br />
|32<br />
|ppn 4, pmePencils 4<br />
|2.88<br />
|1.05<br />
|----<br />
|16<br />
|32<br />
|ppn 4, pmePencils 16<br />
|6.34<br />
|2.31<br />
|----<br />
|16<br />
|32<br />
|ppn 4, pmePencils 20<br />
|6.35<br />
|2.31<br />
|----<br />
|16<br />
|32<br />
|ppn 4, pmePencils 24<br />
|6.24<br />
|2.27<br />
|----<br />
|}<br />
<br />
== Documentation ==<br />
<br />
# [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br />
# [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=User_Python_virtualenv&diff=9409User Python virtualenv2018-08-31T23:32:38Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
== Using Python Virtualenv ==<br />
<br />
Virtual environments (short virtualenv) are a standard in Python to create isolated Python environments. This is useful when certain modules or certain versions of modules are not available in the thee default python environment. First load a python module:<br />
<br />
<source lang="bash"><br />
module load intel/13.1.1 python/2.7.5<br />
</source><br />
<br />
Then install virtualenv:<br />
<br />
<source lang="bash"><br />
pip install virtualenv --user<br />
</source><br />
<br />
Then create a directory for the virtual environments. One can put an virtual environment anywhere, but this directory structure is recommended:<br />
<br />
<source lang="bash"><br />
mkdir ~/.virtualenvs<br />
cd ~/.virtualenvs<br />
</source><br />
<br />
Now we create our first virtualenv called ''myenv'' choose any name you like:<br />
<br />
<source lang="bash"><br />
~/.local/bin/virtualenv myenv<br />
</source><br />
<br />
After that we can activate that virtual environment:<br />
<br />
<source lang="bash"><br />
source ~/.virtualenvs/myenv/bin/activate<br />
</source><br />
<br />
To go back to the normal python installation simply type "deactivate"<br />
As you are in the virtualenv now you can just type ''pip install <required module>'' to install any module in your virtual environment<br />
<br />
===Installing the scientific python suite===<br />
<br />
For many scientific codes the packages ''numpy'', ''scipy'', ''matplotlib'', ''pandas'' and ''ipython'' are used. Except ''scipy'' all of these install very simply in a virtualenv using the ''pip install <package name>''. <br />
<br />
For installing ''scipy'' we first need to install ''numpy'' and then copy scinet's ''site.cfg'' (contains paths to the required libraries for scipy). <br />
So before doing "pip install scipy" do <br />
<br />
<source lang="bash"><br />
cp /scinet/gpc/tools/Python/Python275-shared-intel/lib/python2.7/site-packages/numpy/distutils/site.cfg ~/.virtualenvs/myenv/lib/python2.7/site-packages/numpy/distutils/<br />
</source><br />
<br />
and then just do ''pip install scipy''</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=HPSS-by-pomes&diff=9408HPSS-by-pomes2018-08-31T23:32:18Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
= '''Packing up large data sets and putting them on HPSS''' =<br />
(Pomés group recommendations)<br />
<br />
HPSS has the following limitations:<br />
<br />
Hundreds of thousands of small files can be offloaded rapidly, but take weeks or months to recall<br />
* No individual file can exceed 1 TB<br />
* You must verify the integrity of your data throughout the data preparation and offload process<br />
* With these limitations in mind, we have developed the following protocol for efficiently offloading data from scratch to HPSS.<br />
<br />
<br />
== 1. Identify the subdirectories that contain > 1,000 files. ==<br />
a. Create a directory called DU/ and place the following script in that directory:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# du.sh<br />
for i in $(ls ../); do<br />
n=$(find ../$i |wc -l)<br />
s=$(du -hs ../$i | awk '{print $1}')<br />
echo "$i $n $s"<br />
done > my.du.dirs<br />
</source><br />
<br />
b. chmod +x du.sh<br />
<br />
c. nohup ./du.sh &<br />
(This step may require hours or days to complete)<br />
<br />
d. Now my.du.dirs will contain a listing of the number of files and the total size of each directory.<br />
<br />
e. Identify the directories with many files and copy the DU/ directory there and then run du.sh again. Continue this process until you have a good understanding of which directories actually contain large numbers of files.<br />
<br />
== 2. Create tar files for these directories. ==<br />
a. This should be scripted to ensure that your tarballs are completely written.<br />
<br />
b. Never script the removal of the original files.<br />
<br />
c. Here is an example script:<br />
<source lang="bash"><br />
for i in "dir1 dir2 dir3"; do<br />
tar –cf ${i}.tar ${i}<br />
echo "tar $i returned $?"<br />
done > my.tar.results<br />
</source><br />
<br />
d. Note that the evaluation of $? must be done on the very next command after tar. Even inserting an additional echo statement will break the test.<br />
<br />
e. Once you are sure that the tar command was successful (return code equals zero), you should delete the originals. You should not script the deletion process because a typo in a rm –rf command can be very costly. If you must script a removal, it is best to do it like this:<br />
<br />
mkdir TRASH; for i in $list; do mv $i TRASH; done<br />
<br />
then inspect TRASH and remove it manually. Note that the above command could possibly be costly if you make a mistake since files may get overwritten. <br />
<br />
<br />
== 3. Now upload to HPSS using HSI. ==<br />
You should have less than 10,000 files per TB of uploaded data. If that is not the case, then go back and pack up your data some more before proceeding.<br />
<br />
a. It is recommended that you have a new directory structure in HPSS for your fully packed up data. This is because you may have put other things on HPSS in the past and the creation of a clean directory structure is a good way to denote that this data is the final copy. Here, we use FULL_DATA/<br />
<br />
b. An example of an HPSS offload script follows:<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J offload<br />
#SBATCH --mail-type=ALL<br />
<br />
## scratch files: $SCRATCH/mydata<br />
## HPSS files: $ARCHIVE/FULL_DATA/mydata<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
<br />
/usr/local/bin/hsi -v <<EOF1<br />
mkdir -p FULL_DATA<br />
cd FULL_DATA<br />
cput -Rpuh $SCRATCH/mydata<br />
end<br />
EOF1<br />
<br />
status=$?<br />
if [ ! $status == 0 ];then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
trap - TERM INT<br />
</source><br />
<br />
c. After the above script has completed, check the output to ensure that your transfer was successful. If you had errors, or if it timed out, simply run the script again. If you continue to get the same errors, contact support@scinet.utoronto.ca<br />
<br />
== 4. Check HPSS data against the original ==<br />
Now you must retrieve your data back to scratch so that you can check it against the original copy on scratch, which we have not yet deleted.<br />
<br />
a. Run diskUsage to ensure that you have space in your allocation to recall the data to scratch. If the recall will bring you close to your limit, advise your other group members how much space you will be recalling in case another user is also planning a large data recall.<br />
<br />
b. An example of an HPSS recall script follows:<br />
<source lang="bash"><br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J offload<br />
#SBATCH --mail-type=ALL<br />
<br />
## original scratch files: $SCRATCH/mydata<br />
## HPSS files: $ARCHIVE/FULL_DATA/mydata<br />
## new copy scratch files: $SCRATCH/RETREIVED_MODULES/FULL_DATA/mydata<br />
<br />
mkdir -p $SCRATCH/RETREIVED_MODULES/FULL_DATA<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
<br />
/usr/local/bin/hsi -v <<EOF1<br />
lcd $SCRATCH/RETREIVED_MODULES/FULL_DATA/<br />
cget -Rpuh $ARCHIVE/FULL_DATA/mydata<br />
end<br />
EOF1<br />
<br />
status=$?<br />
if [ ! $status == 0 ];then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
trap - TERM INT<br />
</source><br />
<br />
== 5. Run a md5sum over the entire directory ==<br />
Now that you have the original and the cput/cget copy back from HPSS, run a md5sum over the entire directory. An example of a check is as follows. Run this script under nohup from one of the datamover nodes:<br />
<source lang="bash"><br />
#!/bin/bash<br />
<br />
COMPUTE_DIFFERENCES=1<br />
dir=$(pwd)<br />
WAS=$SCRATCH/mydata<br />
IS=$SCRATCH/RETREIVED_MODULES/FULL_DATA/mydata<br />
if ((COMPUTE_DIFFERENCES)); then<br />
cd $WAS<br />
find . > ${dir}/tmp.was<br />
echo "find on was returned $?"<br />
<br />
cd $IS<br />
find . > ${dir}/tmp.is<br />
echo "find on is returned $?"<br />
<br />
cd $dir<br />
<br />
changes=$(sort tmp.was >a ; sort tmp.is >b; diff a b)<br />
if ((changes!=0)); then<br />
echo "FILES DIFFER! Diff returned $changes"<br />
exit<br />
fi<br />
fi<br />
<br />
for i in $(cat tmp.was); do<br />
if [ -f ${WAS}/${i} ]; then<br />
was=$(md5sum ${WAS}/$i |awk '{print $1}')<br />
is=$(md5sum ${IS}/$i |awk '{print $1}')<br />
same=$(echo $was $is | awk '{if($1==$2) print 1; else print 0}')<br />
if ((same==0)); then<br />
echo "FILES DIFFER -- $i $was $is"<br />
else<br />
echo "OK for $i"<br />
fi<br />
fi<br />
done<br />
</source><br />
<br />
a. when that is done, grep DIFFER on the output (in nohup.out since you ran this script under nohup). Any returned value means there is a problem. Contact scinet support.<br />
<br />
b. If everything was a success, you can delete all of the copies that you recalled from HPSS to scratch. You can also delete your original copy in scratch if you would like as you have a complete copy on scinet HPSS.<br />
<br />
<br />
[[Data Management|BACK TO Data Management]]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=R_Statistical_Package&diff=9407R Statistical Package2018-08-31T23:30:55Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==Running R on the GPC==<br />
[http://www.r-project.org/ R] is powerful statistical and plotting software available on the [[GPC_Quickstart|GPC]] in the [[Software_and_Libraries|module]] R. In fact, there are currently six R modules installed, 2.13.1, 2.14.1, 2.15.1, 3.0.0, 3.0.1 and 3.1.1. While 2.15.1 is the default, we do recommend making the transition to a newer version, which you load by specifying the version number explicitly:<br />
<pre><br />
$ module load intel R/3.0.1<br />
</pre><br />
(The intel module is a prerequesite for the R module). If you will be using Rmpi, you will need to load the openmpi module as well.<br />
<br />
Many optional packages are available for R which add functionality for specific domains; they are available through the [http://cran.r-project.org/mirrors.html Comprehensive R Archive Network (CRAN)]. <br />
<br />
R provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide; there are so many potential optional packages for R people could potentially want, we recommend users who want additional packages to proceed this way. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users package choices don't conflict. <br />
<br />
In general, you can install those that you need yourself in your home directory; eg, <br />
<br />
<pre><br />
$ R <br />
> install.packages("package-name", dependencies = TRUE)<br />
</pre><br />
<br />
will download and compile the source for the packages you need in your home directory under <tt>${HOME}/R/x86_64-unknown-linux-gnu-library/2.11/</tt> (you can specify another directory with a lib= option.) Then take a look at help(".libPaths") to make sure that R knows where to look for the packages you've compiled. Note that you must install packages with logged into a development node as write access to the library folder is not available to a standard node on the cluster. <br />
<br />
Note that during the installation you may get warnings that the packages cannot be installed in e.g. /scinet/gpc/Applications/R/3.0.1/lib64/R/bin/. But after those messages, R should have succeeded in installing the package into your home directory.<br />
<br />
=== Running serial R jobs ===<br />
<br />
As with all serial jobs, if your R computation do not use multiple cores, you should bundle them up so the 8 cores of a nodes are all performing work. Examples of this can be found on the [[User_Serial]] page.<br />
<br />
== Saving images from R in compute jobs ==<br />
<br />
To make use of the graphics capability of R, R insists on having an X server running, even if you're just writing to a file. There is no X server on the compute nodes, and you'd get a message like<br />
<br />
unable to open connection to X11 display ''<br />
<br />
To get around this issue, you can run a 'virtual' X server on the compute nodes by adding the following commands at the start of your job script:<br />
<br />
# Make virtual X server command called Xvfb available:<br />
module load Xlibraries<br />
<br />
# Select a unique display number:<br />
let DISPLAYNUM=$UID%65274<br />
export DISPLAY=":$DISPLAYNUM"<br />
<br />
# Start the virtual X server<br />
Xvfb $DISPLAY -fp $SCINET_FONTPATH -ac 2>/dev/null &<br />
<br />
After this, run R or Rscript as usual. The virtual X server will be running in the background and will get killed which your job is done. Alternatively, you may want to kill it explicitly at the end of you job script using <br />
<br />
# Kill any remaining Xvfb server<br />
pkill -u $UID Xvfb<br />
<br />
== Rmpi (R with MPI) ==<br />
<br />
All the newer R installations on the GPC have Rmpi installed by default using OpenMPI. Be sure to load the OpenMPI module if you wish to use Rmpi.<br />
<br />
=== Installing Rmpi, version 2.13.1 ===<br />
<br />
Version 2.13.1 does not have the Rmpi library as a standard package, which means you have to install it yourself if you are using that version. The same is true if you want to use IntelMPI instead of OpenMPI. <br />
<br />
Installing the Rmpi package can be a bit challenging, since some additional parameters need to be given to the installation, which contain the path to various header files and libraries. These paths differ depending on what [[GPC_MPI_Versions|MPI version]] you are using. <br />
<br />
The various MPI versions on the GPC are loaded with the [[Software_and_Libraries|module]] command. So the first thing to do is to decide what mpi version to use (openmpi or intelmpi), and to type the corresponding "module load" command on the command-line (as well as in your jobs scripts).<br />
<br />
Because the MPI modules define all the paths in environment variables, the following line seem to work for installations of all openmpi versions.<br />
<br />
<pre><br />
> install.packages("Rmpi",<br />
configure.args =<br />
c(paste("--with-Rmpi-include=",Sys.getenv("SCINET_MPI_INC"),sep=""),<br />
paste("--with-Rmpi-libpath=",Sys.getenv("SCINET_MPI_LIB"),sep=""),<br />
"--with-Rmpi-type=OPENMPI"))<br />
</pre><br />
<br />
For intelmpi, you only need to change <tt>OPENMPI</tt> to <tt>MPICH2</tt> in the last line.<br />
<br />
=== Running Rmpi ===<br />
<br />
To start using R with Rmpi, make sure you have all require modules loaded (e.g. <tt>module load intel openmpi R/2.14.1</tt>), then launch it with<br />
<pre><br />
$ mpirun -np 1 R --no-save<br />
</pre><br />
which starts one master mpi process, but starts up the infrastructure to be able to spawn additional processes.<br />
<br />
== Creating an R cluster ==<br />
<br />
The 'parallel' package allows you to use R to launch individual serial subjobs across multiple nodes. This section describes how this is accomplished.<br />
<br />
=== Creating your Rscript wrapper ===<br />
<br />
The first thing to do is create a wrapper for Rscript. This needs to be done because the R module needs to be loaded on all nodes, but the submission script only loads modules on the head node of the job. The wrapper script, let's call it MyRscript.sh, is short:<br />
<pre><br />
#!/bin/bash<br />
<br />
module load intel/13.1.1 R/3.0.1<br />
${SCINET_R_BIN}/Rscript --no-restore "$@"<br />
</pre><br />
The "--no-restore" flag prevents Rscript from loading your "workspace image", if you have one saved. Loading the image causes problems for the cluster.<br />
<br />
Once you've created your wrapper, make it executable:<br />
<pre><br />
$ chmod u+x MyRscript.sh<br />
</pre><br />
Your wrapper is now ready to be used.<br />
<br />
=== The cluster R code ===<br />
The R code which we will run consists of two parts, the code which launches the cluster, and does pre- and post-analysis, and the code which is run on the individual cluster "nodes". Here is some code which demonstrates this functionality. Let's call it MyClusterCode.R.<br />
<pre><br />
<br />
######################################################<br />
#<br />
# worker code<br />
#<br />
<br />
# first define the function which will be run on all the cluster nodes. This is just a test function. <br />
# Put your real worker code here.<br />
testfunc <- function(a) {<br />
<br />
# this part is just to waste time<br />
b <- 0<br />
for (i in 1:10000) {<br />
b <- b + 1<br />
}<br />
<br />
s <- Sys.info()['nodename']<br />
return(paste0(s, " ", a[1], " ", a[2]))<br />
<br />
}<br />
<br />
<br />
######################################################<br />
#<br />
# head node code<br />
#<br />
<br />
# Create a bunch of index pairs to feed to the worker function. These could be parameters,<br />
# or whatever your code needs to vary across jobs. Note that the worker function only <br />
# takes a single argument; each entry in the list must contain all the information <br />
# that the function needs to run. In this example, each entry contains a list which<br />
# contains two pieces of information, a pair of indices.<br />
indexlist <- list()<br />
index <- 1<br />
for (i in 1:10) {<br />
for (j in 1:10) {<br />
indexlist[index] <- list(c(i,j))<br />
index <- index +1<br />
}<br />
}<br />
<br />
<br />
# Now set up the cluster.<br />
<br />
# First load the parallel library.<br />
library(parallel)<br />
<br />
# Next find all the nodes which the scheduler has given to us.<br />
# These are listed in the file which is indicated by the PBS_NODEFILE<br />
# environment variable.<br />
nodefile <- Sys.getenv("PBS_NODEFILE")<br />
hostnames <- readLines(nodefile)<br />
<br />
# Now launch the cluster, using the list of nodes and our Rscript<br />
# wrapper.<br />
cl <- makePSOCKcluster(names = hostnames, rscript = "/path/to/your/MyRscript.sh")<br />
<br />
# Now run the worker code, using the parameter list we created above.<br />
result <- clusterApplyLB(cl, indexlist, testfunc)<br />
<br />
# The results of all the jobs will now be put in the 'result' variable,<br />
# in the order they were specified in the 'indexlist' variable.<br />
<br />
# Don't forget to stop the cluster when you're finished.<br />
stopCluster(cl)<br />
</pre><br />
You can, of course, add any post-processing code you need to the above code.<br />
<br />
=== Submitting an R cluster job ===<br />
You are now ready to submit your job to the GPC queue. The submission script is like most others:<br />
<pre><br />
#!/bin/bash<br />
#PBS -l nodes=3:ppn=8 <br />
#PBS -l walltime=5:00:00<br />
#PBS -N MyRCluster<br />
<br />
cd $PBS_O_WORKDIR<br />
module load intel/13.1.1 R/3.0.1<br />
${SCINET_R_BIN}/Rscript --no-restore MyClusterCode.R<br />
</pre><br />
Be sure to use whatever number of nodes, length of time, etc., is appropriate for your job.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Phi&diff=9406Phi2018-08-31T23:29:26Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:Xeon_phi.jpg|center|250px|thumb]][[Image:NVIDIA-Tesla-K20X.jpg|center|250px|thumb]]<br />
|name=Intel Xeon Phi / NVIDIA Tesla K20 <br />
|installed=April 2013<br />
|operatingsystem= Linux Centos 6.4<br />
|loginnode= gravity01 <br />
|nnodes=1<br />
|rampernode=32 GB<br />
|corespernode=8 with Xeon Phi & K20<br />
|interconnect=DDR Infiniband<br />
|vendorcompilers=nvcc,pgcc,icc,gcc<br />
|queuetype=none<br />
}}<br />
<br />
This is a single test node, for investigating new accelerator technologies. It consists of a single x86_64 node with one 8-core Intel Sandybridge Xeon <br />
E5-2650 2.0GHz CPU with 32GB of RAM. It has a single NVIDIA Tesla K20 GPU with CUDA Capability 3.0 (Kepler) with 2496 CUDA Cores and 5 GB of RAM as well as a single Intel Xeon Phi 3120A with 57 <br />
1.1 GHz cores and 6GB of RAM. The node is interconnected to the rest of the clusters with DDR Infiniband and mounts the regular SciNet GPFS filesystems. <br />
<br />
=== Login ===<br />
<br />
First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>gravity01</tt>'''.<br />
<br />
=== Queue ===<br />
<br />
As this is a single node users are expected to use it in a "friendly" manner as this system is not setup for production<br />
usage, and primarily for investigating new technologies run times are limited to under 4 hours.<br />
To access the node you need to use the queue, similar to the standard ARC and GPC compute nodes,<br />
however with a maximum walltime of 4 hours.<br />
<br />
For an interactive job use<br />
<pre><br />
qsub -l nodes=1:ppn=8,walltime=1:00:00 -q arcX -I<br />
</pre><br />
<br />
== Software ==<br />
<br />
The same software installed on the GPC is available on '''<tt>arcX</tt>''' using the modules framework. <br />
See '''[[GPC_Quickstart#Modules_and_Environment_Variables | here]]''' for full details.<br />
<br />
== NVIDIA Tesla K20 ==<br />
<br />
See the '''[[ Gravity | Gravity ]]''' wiki page for full details of the available CUDA and OpenCL compilers and modules. To<br />
use all the K20 (Kepler) features a minimum of CUDA 5.0 is required. Cuda/6.5 is recommended for the K20.<br />
<br />
=== CUDA ===<br />
<pre><br />
module load gcc/4.8.1 cuda/6.5<br />
</pre><br />
Here, gcc is loaded because it is a prerequisite of the cuda module.<br />
<br />
You will have to let the cuda compiler know about the capabilities of the Kepler graphics card by supplying the flag <br />
<tt>-arch=sm_30</tt> or <tt>-arch=sm_35</tt>.<br />
<br />
=== Driver Version ===<br />
<br />
The current NVIDIA driver version for the K20 is 340.32<br />
<br />
== Xeon Phi ==<br />
<br />
=== Compilers ===<br />
The Xeon Phi uses the standard intel compilers, however requires at least version 13.1<br />
<br />
<pre><br />
module load intel/14.0.0 <br />
</pre><br />
<br />
=== MPI ===<br />
<br />
IntelMPI also has Xeon Phi support<br />
<br />
<pre><br />
module load intelmpi/4.1.1.036<br />
</pre><br />
<br />
'''NOTE''': Be sure to use '''mpiifort''' for compiling native MIC Fortran code as the '''mpif77,mpif90''' scripts ignore the -mmic flags and will produce host only code.<br />
<br />
=== Tools ===<br />
<br />
The Intel Cluters Tools such as vtune amplifier and inspector are available for the Xeon Phi by<br />
loading the following modules.<br />
<br />
<pre><br />
module load inteltools<br />
</pre><br />
<br />
=== OpenCL ===<br />
<br />
OpenCL version 1.2 is available for the Xeon Phi on '''<tt>arcX</tt>'''<br />
<br />
<pre><br />
/opt/intel/opencl<br />
</pre><br />
<br />
=== Direct Access ===<br />
<br />
The Xeon Phi can be accessed directly from the host node by <br />
<br />
<pre><br />
ssh mic0<br />
</pre><br />
<br />
=== Shared Filesystem ===<br />
<br />
The host node '''arc09''' mounts the standard SciNet filesystems, i.e. $HOME and $SCRATCH, however to share<br />
files between the host and Xeon Phi use /localscratch/$HOME which shows up as $HOME on "mic0".<br />
<br />
=== Useful Links ===<br />
<br />
[http://software.intel.com/en-us/articles/building-a-native-application-for-intel-xeon-phi-coprocessors Building Native for MIC ] <br />
<br />
[http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide#mic TACC Stampede MIC Info ]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Using_Signals&diff=9405Using Signals2018-08-31T23:28:56Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==General==<br />
In unix/linux one can send ''signals'' to a program or a script. Some<br />
common signals are:<br />
{|<br />
! {{Hl2}}|Signal !! !! !! !! {{Hl2}}| Meaning !! !! !! !! {{Hl2}}| Can be trapped?<br />
|-<br />
| HUP || || || || Terminal log-out || || || ||Yes<br />
|-<br />
| INT || || || || Interrupt signal (Ctrl-C is pressed) || || || ||Yes<br />
|- <br />
| TERM || || || || Termination of the process (as if 'kill' was called) || || || || Yes<br />
|- <br />
| KILL || || || || Kill the process (as if 'kill -9' was called) || || || || No<br />
|- <br />
| STOP || || || || Suspend the process || || || ||No<br />
|- <br />
| CONT || || || || Resume the process || || || ||Yes<br />
|- <br />
| USR1 || || || || Unspecified user signal (can be given with 'kill -USR1') || || || ||Yes<br />
|-<br />
| USR2 || || || || Unspecified user signal (can be given with 'kill -USR2') || || || ||Yes<br />
|}<br />
<br />
From the node on which a a process is running, signals can be given using <br />
<pre> $ kill -[SIGNAL] [PID]<br />
</pre> <br />
On the GPC, you can pass a signal to a running job using<br />
<pre><br />
$ qsig -s [SIGNAL] [JOBID]<br />
</pre><br />
So for example, you could suspend the process running under job id 314159 by <tt>qsig -s STOP 314159</tt>, and resume it using <tt>qsig -s CONT 314159</tt> (note that the wall clock keeps ticking!).<br />
<br />
For qsig to work for signals whose default action is to kill your job (such as HUP, INT, TERM, and, perhaps surprisingly, USR1 and USR2), your job script should either trap these explicitly (see below), or contain a line <br />
<pre><br />
trap : [LIST-OF-SIGNALS-TO-BE-PASSED] <br />
</pre><br />
(e.g. <tt>trap : USR1 USR2</tt>) before the application that traps the signal is started.<br />
Note that he spaces surrounding the colon are essential here, because the colon is actually a bash command (one that does nothing).<br />
<br />
Signals can also be given by the system. One important instance of system signals is that '''on SciNet, a TERM signal is sent to a jobs whose requested time is over,''' after which the job has about 2 minutes to clean up. <br />
<br />
With the techniques explained below, a program or script can be set up to listen for a particular kind of signal, except for KILL and STOP (which always kill and suspend a process, respectively). When it receive a signal of that kind, its execution is interrupted and a call is made to a function, specified<br />
earlier in the program. It is up to the program or script what this signal handling function does but it is a good idea to make the action appropriate for the event that triggers the signal. For instance, a TERM signal should be handled as a request to terminate the application. <br />
The user signals USR1 and USR2 do not have a pre-designated meaning and can be used for application-specific actions such as checkpointing.<br />
<br />
==Trapping signals in bash scripts==<br />
<br />
To trap signals in a bash script, one has to bind a specific<br />
signal to a command or function with the <tt>trap</tt><br />
command. For example,<br />
<source lang="bash"> <br />
#!/bin/bash<br />
trap "echo Term was trapped.; exit" TERM<br />
for ((i=0;i<60;i++)) do<br />
echo $i<br />
sleep 1<br />
done<br />
</source><br />
Running this script in the background and sending it a TERM command ('<tt>kill -TERM [pid]</tt>') will cause the message '<tt>Term was trapped.</tt>' to be printed. Notes:<br />
* The trapped command has to use '<tt>exit</tt>' explicitly to stop the script's execution. <br />
* The '<tt>sleep</tt>' command in bash cannot be interrupted, which is why the script contains a succession of 1 second '<tt>sleep</tt>'s.<br />
<br />
Another useful example of signal trapping in a bash script is given on the wiki page about <br />
[[User_Ramdisk | using ramdisk]].<br />
<br />
==Trapping signals in C==<br />
<br />
To trap signals in a c program, one has to include the<br />
<tt>signal.h</tt> header file:<br />
<source lang="c"><br />
#include <signal.h><br />
</source><br />
and provide a signal handler function<br />
<source lang="c"><br />
void term_trap(int sig) <br />
{<br />
/* do something */<br />
}<br />
</source><br />
which is linked to the specific signal somewhere as follows:<br />
<source lang="c"><br />
signal(SIGTERM, term_trap);<br />
</source><br />
Note that the names of signals are prepended with "<tt>SIG</tt>" in <br />
<tt>signal.h</tt>.<br />
<br />
A minimal example:<br />
<source lang="c"><br />
#include <stdio.h><br />
#include <signal.h><br />
#include <unistd.h><br />
<br />
void term_trap(int sig) <br />
{<br />
printf("Term was trapped.\n");<br />
}<br />
<br />
int main() <br />
{<br />
signal(SIGTERM, term_trap);<br />
sleep(60);<br />
}<br />
</source><br />
Note that the names of the signals are prepended with <tt>SIG</tt> in C.<br />
To test this program:<br />
* save the above code as <tt>sigex.c</tt><br />
* compile: <tt>icc -O3 -xHost sigex.c -o sigex</tt><br />
* run: <tt>sigex&</tt> <br />
* note the pid (process idenitfier) <br />
* you can then give the process the term signal with <tt>kill -TERM [pid]</tt>.<br />
<br />
==Trapping signals in C++==<br />
<br />
The same method as for a C program works in C++, except the header file is '<tt>csignal</tt>'. For example:<br />
<source lang="c"><br />
#include <iostream><br />
#include <csignal><br />
#include <unistd.h><br />
<br />
void term_trap(int sig) <br />
{<br />
std::cout << "Term was trapped.\n";<br />
}<br />
<br />
int main() <br />
{<br />
signal(SIGTERM, term_trap);<br />
sleep(60);<br />
}<br />
</source><br />
<br />
==Trapping signals in Fortran==<br />
<br />
Unfortunately, signal handling in fortran is not standard, and different compilers have different ways of doing this. <br />
Below, examples are given for the three fortran compilers available on SciNet: ifort, gfortran, and xlf. These examples were designed such that even though the ways to register the handler differ, the signal handlers themselves are always the same.<br />
<br />
'''''ifort'''''<br />
<source lang="fortran"><br />
C IFORT SIGNAL TRAPPING EXAMPLE<br />
USE IFPORT<br />
EXTERNAL TRAP_TERM<br />
INTEGER TRAP_TERM<br />
INTEGER ERR<br />
ERR = SIGNAL (SIGTERM, TRAP_TERM, -1)<br />
CALL SLEEP (60)<br />
END<br />
C SIGNAL HANDLER FUNCTION<br />
FUNCTION TRAP_TERM (SIG_NUM)<br />
INTEGER TRAP_TERM<br />
INTEGER SIG_NUM<br />
PRINT *, "Term was trapped."<br />
TRAP_TERM = 1<br />
END<br />
</source><br />
<br />
'''''gfortran'''''<br />
<source lang="fortran"><br />
C GFORTRAN SIGNAL TRAPPING EXAMPLE<br />
INTRINSIC SIGNAL<br />
INTEGER SIGTERM<br />
PARAMETER (SIGTERM = 15)<br />
EXTERNAL TRAP_TERM<br />
CALL SIGNAL (SIGTERM, TRAP_TERM)<br />
CALL SLEEP (60)<br />
END<br />
C SIGNAL HANDLER FUNCTION<br />
FUNCTION TRAP_TERM (SIG_NUM)<br />
INTEGER TRAP_TERM<br />
INTEGER SIG_NUM<br />
PRINT *, "Term was trapped."<br />
TRAP_TERM = 1<br />
END<br />
</source><br />
Note: the SIGNAL function is broken in the official gfortran versions 4.6.0 and 4.6.1, but this bug has been fixed in SciNet's gcc modules.<br />
<br />
'''''xlf'''''<br />
<source lang="fortran"><br />
C XLF SIGNAL TRAPPING EXAMPLE<br />
INCLUDE 'fexcp.h'<br />
INTEGER SIGTERM<br />
PARAMETER (SIGTERM = 15)<br />
EXTERNAL TRAP_TERM<br />
CALL SIGNAL (SIGTERM, TRAP_TERM)<br />
CALL SLEEP (60)<br />
END<br />
C SIGNAL HANDLER FUNCTION<br />
FUNCTION TRAP_TERM (SIG_NUM)<br />
INTEGER TRAP_TERM<br />
INTEGER SIG_NUM<br />
PRINT *, "Term was trapped."<br />
TRAP_TERM = 1<br />
END<br />
</source><br />
--Rzon 30 June 2010</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Performance_And_Debugging_Tools:_GPC&diff=9404Performance And Debugging Tools: GPC2018-08-31T23:28:19Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==Memory Profiling ==<br />
<br />
===PurifyPlus===<br />
This is IBM/Rational's set of tools for memory profiling and code testing. Specifically, the tools are: purify, purecov and quantify.<br />
<br />
In order to gain access to these tools, you need to source the script that defines them to your shell:<br />
<br />
<tt> source /scinet/gpc/Rational/purifyplus_setup.sh </tt><br />
<br />
The documentation for getting started with these tools is in the directory:<br />
<br />
<tt> /scinet/gpc/Rational/releases/PurifyPlus.7.0.1/docs/pdf </tt><br />
<br />
or here: [http://support.scinet.utoronto.ca/Manuals/PurifyPlusGettingStarted_en.pdf PurifyPlus Getting Started].<br />
<br />
===Valgrind===<br />
<br />
Valgrind http://valgrind.org/ is a suite of tools for debugging and profiling programs especially<br />
useful for finding memory problems, such as memory leaks and segfaults. To use it on the GPC <br />
you must first load the module valgrind.<br />
<br />
<tt> module load valgrind </tt><br />
<br />
In serial valgrind can simply by run as follows, with no need to recompile your binary<br />
<br />
<tt>valgrind --tool=memcheck ./a.out </tt><br />
<br />
there are many useful flags such as --leak-check=yes and --show-reachable=yes that<br />
can be found by running valgrind --help on consulting the man pages.<br />
<br />
Valgrind can also be used in parallel with MPI as well, http://valgrind.org/docs/manual/mc-manual.html#mc-manual.mpiwrap<br />
in a similar fashion, however a library needs to preloaded first.<br />
<br />
<tt> LD_PRELOAD=${SCINET_VALGRIND_LIB}/valgrind/libmpiwrap-amd64-linux.so mpirun -np 2 valgrind --tool=memcheck ./a.out </tt><br />
<br />
Besides being a extremely good at [[Performance_And_Debugging_Tools:_GPC#Memory_Profiling | finding memory problems]], [http://valgrind.org/ valgrind] comes with a tool called [http://valgrind.org/docs/manual/cg-manual.html cachegrind] which can find cache use problems in your code; its use is described in our [[Introduction_To_Performance#cachegrind_.28Memory_use_analysis:_GPC.29 | Intro To Performance]].<br />
<br />
==Debugging==<br />
<br />
===gdb===<br />
<br />
[http://www.gnu.org/software/gdb/ GDB] is solid source-language level serial debugger. [https://support.scinet.utoronto.ca/wiki/images/2/27/Snugdebug.pdf This SNUG TechTalk from Nov 2010] introduces debugging with gdb. gdb will debug code compiled with both intel and gcc compilers. Use <tt>module load gdb</tt> to ensure you are using the most recent version of gdb.<br />
<br />
<tt>gdb -tui</tt> launches a text-based interface that functions a bit like a gui for debugging code, and can be handy.<br />
<br />
===idbc===<br />
<br />
idb is the Intel debugger; it will work with code compiled with both intel and gcc compilers. Use <tt>module load intel</tt> to ensure the debugger is in your path. The command-line debugger is launched by invoking <tt>idbc</tt>; launching <tt>idb</tt> will start a graphical, [http://www.eclipse.org/ Eclipse]-based environment, which can be quite slow over a network connection. If you want nonetheless want to use the graphical interface, ensure that you have Java in your enviroment; ues <tt>module load java</tt>.<br />
<br />
===ddd===<br />
<br />
[http://www.gnu.org/software/ddd/ Data Display Debugger (ddd)] is a graphical wrapper to the gdb or idb debuggers. It is quite handy for debugging scientific programs as it has fairly sophisticated data-plotting features where array variables can be plotted dynamically as the program progresses. As with <tt>idb</tt>, however, using the graphical interface over a network connection can be quite laggy and slow. Use <tt>module load ddd</tt> to load ddd into your environment.<br />
<br />
===ddt===<br />
<br />
ddt is Allinea's graphical parallel debugger, in the ddt module. Highly recommended! Use <tt>module load ddt</tt> to load ddt into your environment.<br />
<br />
==Performance Profiling==<br />
<br />
===gprof===<br />
<br />
[http://sourceware.org/binutils/docs-2.19/gprof/index.html gprof] is a very useful tool for finding out where a program is spending its time; its use is described in our [[Introduction_To_Performance#gprof_.28profiling:_everywhere.29 | Intro To Performance]].<br />
<br />
===Open|SpeedShop===<br />
<br />
[http://www.openspeedshop.org OpenSpeedshop] is a tool for performing sampling and MPI tracing of a program on the GPC. It's use is outlined in our [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | Intro to Performance]]. It is right now compiled for use only with openmpi and gcc-4.4; to use it,<br />
<br />
<pre><br />
module load gcc openmpi openspeedshop<br />
</pre><br />
<br />
To run a sampling experiment with the program, run the program through openspeedshop then use the same program to view the results:<br />
<br />
<pre><br />
$ openss -f "./a.out" pcsamp<br />
[program runs as usual, outputs extra performance data at the end]<br />
$ openss -f [experimentname].openss<br />
</pre><br />
<br />
It can also trace MPI calls and give detailed statistics about the time spent in MPI routines by process number; to test this on a run done on one of the devel nodes,<br />
<br />
<pre><br />
$ module load gcc openmpi/1.3.2-gcc-v4.4.0-ofed openspeedshop<br />
$ openss -f "mpirun -np 6 ./a.out" mpit<br />
[program runs as usual, outputs extra performance data at the end]<br />
$ openss -f [experimentname].openss<br />
</pre><br />
<br />
this can also be used to perform tracing on batch jobs by ensuring <tt>module load gcc openmpi/1.3.2-gcc-v4.4.0-ofed openspeedshop</tt> is in your <tt>.bashrc</tt> and mpirun-ing the program as shown above.<br />
<br />
<br />
<br />
===Scalasca===<br />
<br />
[http://www.scalasca.org/ Scalasca] is a sophisticated tool for analyzing performance and finding common performance problems. We describe it in our [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Intro to Performance]].</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Ansys&diff=9403Ansys2018-08-31T23:27:31Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
The [http://www.ansys.com/ Ansys] engineering simulation tools are installed on SciNet.<br />
<br />
=Getting a license=<br />
<br />
Licenses are provided by [http://www.cmc.ca/ CMC Microsystems]. Canadian students and faculty<br />
can register at https://www.cmc.ca/en/MyAccount/GetAccount.aspx<br />
<br />
Once you have an account, you have to contact CMC and tell them you want to use the Ansys tools on SciNet,<br />
and give them your SciNet username.<br />
<br />
=Ansys 18.1=<br />
<br />
module load ANSYS/18.1<br />
<br />
Programs available:<br />
<br />
* fluent<br />
* ansysedt<br />
* mapdl<br />
* ...<br />
<br />
==Running Fluent==<br />
<br />
Example submission script for a job running on 4 nodes, with max walltime of 24 hours:<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#PBS -l nodes=4:ppn=8,walltime=24:00:00<br />
<br />
module load ANSYS/18.1 extras<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
INPUTFILE=input.jou<br />
fluent 2ddp -t "$PBS_NP" -cnf="$PBS_NODEFILE" -mpi=intel -pib -pcheck -g -i "$INPUTFILE"<br />
</source></div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Niagara&diff=9402Niagara2018-08-31T23:27:13Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
Information on the Niagara supercomputer can be found on<br />
https://docs.computecanada.ca/wiki/Niagara/en.<br />
<br />
Les informations sur le superordinateur Niagara se trouvent sur<br />
https://docs.computecanada.ca/wiki/Niagara/fr.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=GPU_Devel_Nodes&diff=9401GPU Devel Nodes2018-08-31T23:26:46Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
<span style="color:#772222">The ARC GPU have been decommisioned. The head node, arc01, is still up, however for GPU computations, users are encouraged to move to the [[Gravity]] cluster. For visualization, new [[Visualization Nodes]] are being setup.</span></div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Using_MySQL_on_the_GPC&diff=9400Using MySQL on the GPC2018-08-31T23:26:09Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
''TechTalk by Ramses van Zon (SciNet)''<br />
<br />
''Oct 16, 2013''<br />
<br />
MySQL is a relational database management system (RDBMS) that runs as a server providing multi-user access to a number of databases. It is often used by free-software/open-software projects requiring RDBMS. Running a single relational database version for all of SciNet is not a good fit, as the needs of our many users could easily conflict and uptime of the server could not always be guarranteed.<br />
<br />
It is however possible to setup MySQL in user space, using a few homebrewed scripts to help resolve some of the common issues with running multiple database servers. In this TechTalk, we will show how you can use MySQL in user space on the General Purpose Cluster for short tests on the devel node, within a single node compute job, or even spanning multiple nodes.<br />
<br />
''Note: this is the content of the slides used in the TechTalk. This is also available in [[Media:Ttmysql.pdf|pdf format]].''<br />
<br />
=== Data storage types ===<br />
<br />
Data in all shapes and forms:<br /> <br /><br />
<br />
* Standard binary data (hdf5, netcdf, grib2, jpg, pdb, …)<br />
* Standard/semi-structured text data (xml, json, csv, fasta)<br />
* Proprietary binary data format<br />
* Unstructured text data (logs, output and error files)<br />
* '''Databases'''<br />
<br />
----<br />
=== Databases ===<br />
<br />
'''Database:''' an organized collection of data for storing, managing and retrieving information<br />
<br />
* Often back-end to a website or application.<br />
* Typically structured as tables.<br />
* Each row expresses a ''relation'' between columns.<br />
* SQL: Structured Query Language.<br />
* Implementations often have access control on databases and tables.<br />
<br />
Exception to this structure: ''NoSQL''. Not in this TechTalk.<br />
<br />
<br />
----<br />
<br />
=== Why use a relational database? ===<br />
<br />
* You like/are familiar with SQL.<br />
* Application relies on a RDBMS.<br />
* You want to be able express complex queries, such as:<br />
<br />
<blockquote>List the directories of stored simulations in which the pressure was between 1 and 2 atm and the number of water molecules was between 4,000 and 50,000.<br />
<br />
<source lang="bash">SELECT B.directory FROM simulations_attributes A <br />
INNER JOIN simulation_paths B WHERE<br />
A.simulation_id=B.simulation_id AND <br />
A.pressure>=1 AND A.pressure<=2 AND <br />
A.molecules>=4000 AND A.molecules<=50000;</source></blockquote><br />
Note that you rarely store your binary data in the DB,<br />but you could store paths and filenames with metadata.<br />
<br />
----<br />
<br />
=== Some Relational Database Management Systems ===<br />
<br />
* Commercial:<br />
** Oracle Database<br />
** IBM DB2<br />
** Microsoft SQL Server<br />
** SAP Sybase<br />
* Open Source:<br />
** postgreSQL<br />
** SQLite<br />
** '''MySQL'''<br />
<br />
----<br />
<br />
=== What is MySQL? ===<br />
<br />
* Open-source RDBMS<br />
* Was owned by MySQL AB, now by Oracle<br />
* Often used by Free-software/open-software projects requiring RDBMS<br />
* Command-line interface (<code>mysql</code>)<br />
* Interfaces for C, python, php, perl, R, …<br />
* Third party graphical interfaces<br />
<br />
----<br />
<br />
=== Typical MySQL Deployment ===<br />
<br />
* Database server runs as root on some server<br />
* A ’root’ user can setup privileges on the databases and tables<br />
* Application/web service runs on client (sometimes server=client)<br />
* and contacts the server as a non-root user through a port or socket.<br />
<br />
----<br />
<br />
=== Why This Setup Doesn’t Work on SciNet: ===<br />
<br />
* Don’t want to maintain a database server accessible to 1000+ users:<br />
** '''Scalability:''' Many nodes may try to access DB<br />
** '''Conflicts:''' Users may run same application, access the same databases/tables.<br />
** '''Security:''' What if privileges are not set correctly?<br />
** '''Stability:''' If DB goes down, or becomes slow, all jobs go with it.<br />
** '''File systems:''' RDBMS may not play nice with file system (iops-heavy).<br />
* Users cannot start root processes or setup privileges.<br />
<br />
'''Luckily it is possible to install MySQL in user space.'''<br />
<br />
----<br />
<br />
=== User space MySQL Deployment ===<br />
<br />
* Database server runs as user on some machine.<br />
* Creates its own ’root’ user, sets up privileges on databases and tables<br />
* ''This is a database user called ’root’, which does not have, and should not, have anything at all to do with the linux root user.''<br />
* Application runs on another or the same machine<br />
* Port numbers and sockets must be assigned properly to avoid conflicts with other users’ databases''<br />
* Application contacts this server as a non-’root’ user through the properly assigned port or socket.<br />
<br />
----<br />
<br />
=== SciNet’s MySQL Module on the GPC ===<br />
<br />
* MySQL user deployment is a bit of a pain, and would mean a lot of duplication of effort.<br />
* So we’ve done that already, and wrote a few scripts to help you.<br />
* They are in the ’mysql’ module.<br />
<br />
<br />'''Let’s demonstrate how this works…'''<br />
<br />
----<br />
<br />
=== Basic Example ===<br />
<br />
<source lang="bash">$ module load mysql<br />
$ mysqlsetup<br />
Creating directory (tree) $SCRATCH/.mysql-5.6.12/default for configuration files<br />
Creating directory $SCRATCH/mysqldata/default for databases<br />
Creating local configuration<br />
Initializing local mysql database<br />
Enter mysql root password:<br />
Confirm mysql root password: <br />
Starting mysql service<br />
131011 15:05:31 mysqld_safe Logging to '$SCRATCH/mysqldata/default/gpc-f103n084.err'.<br />
131011 15:05:31 mysqld_safe Starting mysqld daemon with databases from $SCRATCH/mysqldata/default<br />
Setting password<br />
Cleaning up<br />
Stopping mysql service<br />
131011 15:05:36 mysqld_safe mysqld from pid file $SCRATCH/mysqldata/default/gpc-f103n084.pid ended<br />
$</source><br />
<br />
Next, starting the server:<br />
<source lang="bash">$ module load mysql<br />
$ mysqlsetup<br />
...<br />
$ mysqlstart<br />
131011 15:09:39 mysqld_safe Logging to '$SCRATCH/mysqldata/default/gpc-f103n084.err'.<br />
131011 15:09:39 mysqld_safe Starting mysqld daemon with databases from $SCRATCH/mysqldata/default<br />
$ mysql -u root -p<br />
Enter password:<br />
Welcome to the MySQL monitor. Commands end with ; or \g.<br />
...<br />
mysql> \q<br />
$ mysqlstop<br />
131011 15:18:34 mysqld_safe mysqld from pid file $SCRATCH/mysqldata/default/gpc-f103n084.pid ended<br />
$ </source><br />
<br />
<br />
'''What happened?'''<br />
<br />
* Stuff was setup on scratch, so we can use this on compute nodes too.<br />
* Setup root password<br />
* Directories created:<br />
** <code>$SCRATCH/.mysql-5.6.12/default</code>: contains configuration files<br />
** <code>$SCRATCH/mysqldata/default</code>: contains database files<br />
* Configuration is such that non-default sockets and ports are used: multiple users can use this module simultaneously on devel nodes.<br />
* Tip: do not use the root user, create a new user.<br />
<br />
----<br />
<br />
=== Using Profiles ===<br />
<br />
'''Note <code>default</code> in the above?'''<br />
<br />
* It’s possible to have other configurations, using different directories. This is required if you want to run a database server for each job.<br />
* Each setup is called a ’profile’.<br />
* Each profile has its own configuration and data directory.<br />
* Each profile has its own root password.<br />
* Data directory can even be on ramdisk!<br />(but you’ll have to copy it if you want to save your results).<br />
<br />
----<br />
<br />
=== Profile Example ===<br />
<br />
<source lang="bash">$ module load mysql<br />
$ mysqlsetup -P compute<br />
$ mysqlstart -P compute<br />
$ export MYSQL_HOME=$MYSQL_HOMES/compute<br />
$ mysql -u root -p<br />
mysql> CREATE DATABASE computedb<br />
mysql> GRANT ALL PRIVILEGES ON computedb.* TO computeuser<br />
mysql> FLUSH PRIVILEGES<br />
mysql> \q<br />
$ mysqlstop -P compute</source><br />
<br />
----<br />
<br />
=== Ramdisk Example ===<br />
<br />
<source lang="bash">$ mkdir -p /dev/shm/rzon/mysqldata<br />
$ mysqlsetup -P ramdisk -D /dev/shm/rzon/mysqldata<br />
$ mysqlstart -P ramdisk<br />
$ mysqlstart -P default<br />
$ mysqlstatus<br />
PROFILE PID<br />
ramdisk 5243<br />
default 5740<br />
$ export MYSQL_HOME=$MYSQL_HOMES/ramdisk<br />
$ mysql -u root -p<br />
$ export MYSQL_HOME=$MYSQL_HOMES/default<br />
$ mysql -u root -p<br />
$ mysqlstop -P ramdisk<br />
$ cd /dev/shm/rzon/mysqldata<br />
$ tar cf $SCRATCH/ramdiskdb.tar .</source><br />
<br />
Next run can just do:<br />
<br />
<source lang="bash">$ module load mysql<br />
$ cd /dev/shm/rzon/mysqldata<br />
$ tar xpf $SCRATCH/ramdiskdb.tar <br />
$ mysqlstart -P ramdisk<br />
$ export MYSQL_HOME=$MYSQL_HOMES/ramdisk<br />
$ ...<br />
$ mysqlstop -P ramdisk<br />
$ tar cf $SCRATCH/ramdiskdb.tar /dev/shm/rzon/mysqldata</source><br />
Note that if there are several runs at the same time, you’ll have to have different tar files on the file system and may have to figure out how to merge databases.<br />
<br />
----<br />
<br />
=== Multi-node Example ===<br />
<br />
A script that uses the mysql database but does not start it<br />
<br />
<source lang="bash">#!/bin/bash<br />
# file: compute.sh<br />
module load mysql<br />
export MYSQL_HOME=$MYSQL_HOMES/compute<br />
export mysqlhost=$1<br />
export thiscase=$2<br />
mysql -h $mysqlhost -u computeuser computedb ... $thiscase ...<br />
</source><br />
<br />
<source lang="bash">#!/bin/bash<br />
#PBS -l nodes=2:ppn=8,walltime=24:00:00<br />
cd $PBS_O_WORKDIR<br />
module load gnu-parallel/20130422 mysql<br />
mysqlstart -P compute<br />
export mysqlhost=$(hostname)<br />
parallel -j2 --slf $PBS_NODEFILE --workdir $PWD <<EOF<br />
compute.sh $mysqlhost 1<br />
compute.sh $mysqlhost 2<br />
compute.sh $mysqlhost 3<br />
...<br />
EOF<br />
mysqlstop -P compute<br />
#end of job script</source><br />
<br />
<br />
<source lang="bash">$ qsub jobscript</source><br />
<br />
----<br />
<br />
=== Summary ===<br />
<br />
* The <code>mysql</code> module allows you to run mysql server as a user.<br />
* Without bothering other users.<br />
* Can run multiple mysql servers using profiles (must use different directories).<br />
* Can run on ramdisk.<br />
* Can serve a multinode job<br />
* Scripts from the mysql module:<br />
** <code>mysqlsetup</code><br />
** <code>mysqlstart</code><br />
** <code>mysqlstatus</code><br />
** <code>mysqlstop</code><br />
<br />
----</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Accelerator_Research_Cluster&diff=9399Accelerator Research Cluster2018-08-31T23:25:00Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:Tesla S2070 3qtr.gif|center|200px|thumb]]<br />
|name=Accelerator Research Cluster (ARC)<br />
|installed=June 2010, April 2011<br />
|operatingsystem= Linux (Centos 6.2)<br />
|loginnode= arc01 (from login.scinet)<br />
|nnodes=8(x86)+4x4(GPU)+14(Cell)<br />
|interconnect=DDR Infiniband<br />
}}<br />
The Accelerator Research Cluster (ARC) is a technology evaluation cluster with a combination of 14 IBM PowerXCell 8i "Cell" nodes and 8 Intel x86_64 "Nehalem" nodes containing 16 NVIDIA M2070 GPUs.<br />
The QS22's each have two 3.2GHz "IBM PowerXCell 8i CPU's, where each CPU has 1 Power Processing Unit (PPU) and 8 Synergistic Processing Units (SPU), and<br />
32GB of RAM per node. The Intel nodes have two 2.53GHz 4core Xeon X5550 CPU's with 48GB of RAM per node each containing two NVIDIA M2070 (Fermi) GPU's each with 6GB of RAM.<br />
<br />
Please note that this cluster is not a production cluster and is only accessible to selected users. <br />
<br />
===Login===<br />
<br />
First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>arc01</tt>''' which <br />
is currently the gateway/devel node for this cluster.<br />
<br />
==Compile/Devel/Compute Nodes==<br />
<br />
=== [[ Cell_Devel_Nodes | Cell ]] ===<br />
You can log into any of 12 nodes '''<tt>blade[03-14]</tt>''' directly to compile/test/run Cell<br />
specific or OpenCL codes.<br />
<br />
See the [[ Cell_Devel_Nodes | Cell Devel Info ]] page for Cell specific details.<br />
<br />
=== [[ GPU_Devel_Nodes | GPU ]]===<br />
You can log into the devel node '''<tt>arc01</tt>''' directly to compile and<br />
interactively test, and from there submit jobs to the other 7 x86_64/GPU nodes.<br />
<br />
See the [[ GPU_Devel_Nodes | GPU Devel Info ]] page for GPU specific details.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=MemP&diff=9398MemP2018-08-31T23:24:28Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
memP is a Lawrence Livermore National Labs (LLNL) developed, light weight, parallel heap profiling library. Its primarily designed to identify the heap allocation that causes an MPI task to reach its memory in use high water mark (HWM).<br />
<br />
== memP Reports ==<br />
<br />
'''Summary Report:''' Generated from within MPI_Finalize, this report describes the memory HWM of each task over the run of the application. This can be used to determine which task allocates the most memory and how this compares to the memory of other tasks.<br />
<br />
'''Task Report:''' Based on specific criteria, a report can be generated for each task, that provides a snapshot of the heap memory currently in use, including the amount allocated at specific call sites.<br />
<br />
==Using memP==<br />
<br />
Load the memp Module<br />
<br />
<pre><br />
module load memP<br />
</pre><br />
<br />
Compile with the recommended BG/Q flags and link your application with the required libraries:<br />
<br />
<pre><br />
-Wl,-zmuldefs ${SCINET_LIB_MEMP}<br />
</pre><br />
<br />
Examples:<br />
<pre><br />
mpixlc -g -Wl,-zmuldefs -o myprog myprog.c -L/usr/local/tools/memP/lib -lmemP<br />
mpixlf77 -g -Wl,-zmuldefs -o myprog myprog.f -L/usr/local/tools/memP/lib -lmemP <br />
</pre><br />
<br />
Then run your MPI application as usual and you will see an memP header and traile it sends to stdout, as well as the output file generated at the end of the run.<br />
<br />
==Output Options==<br />
<br />
See http://memp.sourceforge.net/ for full details.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=User_Codes&diff=9397User Codes2018-08-31T23:23:58Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
__FORCETOC__<br />
<br />
==Astrophysics==<br />
<br />
===Athena (explicit, uniform grid MHD code)===<br />
<br />
[[Image:StrongScalingAthenaGPC.png|thumb|right|320px|Athena scaling on GPC with OpenMPI and MVAPICH2 on GigE, and OpenMPI on InfiniBand]]<br />
<br />
[http://www.astro.princeton.edu/~jstone/athena.html Athena] is a straightforward C code which doesn't use a lot of libraries so it is pretty straightforward to build and compile on new machines. <br />
<br />
It encapsulates its compiler flags, etc in an <tt>Makeoptions.in</tt> file which is then processed by <tt>configure</tt>. I've used the following additions to <tt>Makeoptions.in</tt> on TCS and GPC:<br />
<br />
<source lang="make"><br />
ifeq ($(MACHINE),scinettcs)<br />
CC = mpcc_r<br />
LDR = mpcc_r<br />
OPT = -O5 -q64 -qarch=pwr6 -qtune=pwr6 -qcache=auto -qlargepage -qstrict<br />
MPIINC =<br />
MPILIB =<br />
CFLAGS = $(OPT)<br />
LIB = -ldl -lm<br />
else<br />
ifeq ($(MACHINE),scinetgpc)<br />
CC = mpicc<br />
LDR = mpicc<br />
OPT = -O3<br />
MPIINC =<br />
MPILIB =<br />
CFLAGS = $(OPT)<br />
LIB = -lm<br />
else<br />
...<br />
endif<br />
endif<br />
</source><br />
It performs quite well on the GPC, scaling extremely well even on a strong scaling test out to about 256 cores (32 nodes) on Gigabit ethernet, and performing beautifully on InfiniBand out to 512 cores (64 nodes). <br />
<br />
<br />
-- [[User:Ljdursi|ljdursi]] 19:20, 13 August 2009 (UTC)<br />
<br />
===FLASH3 (Adaptive Mesh reactive hydrodynamics; explict hydro/MHD)===<br />
<br />
[[Image:weak-scaling-example.png|thumb|right|320px|Weak scaling test of the 2d sod problem on both the GPC and TCS. The results are actually somewhat faster on the GPC; in both cases (weak) scaling is very good out at least to 256 cores]]<br />
<br />
[http://flash.uchicago.edu FLASH] encapsulates its machine-dependant information in the <tt>FLASH3/sites</tt> directory. For the GPC, you'll have to<br />
<pre><br />
module load intel<br />
module load openmpi<br />
module load hdf5/184-p1-v16-openmpi<br />
</pre><br />
<br />
and with that, the following file (<tt>sites/scinetgpc/Makefile.h</tt>) works for me:<br />
<source lang="make"><br />
## Must do module load hdf5/183-v16-openmpi<br />
HDF5_PATH = ${SCINET_HDF5_BASE}<br />
ZLIB_PATH = /usr/local<br />
<br />
#----------------------------------------------------------------------------<br />
# Compiler and linker commands<br />
#<br />
# We use the f90 compiler as the linker, so some C libraries may explicitly<br />
# need to be added into the link line.<br />
#----------------------------------------------------------------------------<br />
<br />
## modules will put the right mpi in our path<br />
FCOMP = mpif77<br />
CCOMP = mpicc<br />
CPPCOMP = mpiCC<br />
LINK = mpif77<br />
<br />
#----------------------------------------------------------------------------<br />
# Compilation flags<br />
#<br />
# Three sets of compilation/linking flags are defined: one for optimized<br />
# code, one for testing, and one for debugging. The default is to use the <br />
# _OPT version. Specifying -debug to setup will pick the _DEBUG version,<br />
# these should enable bounds checking. Specifying -test is used for <br />
# flash_test, and is set for quick code generation, and (sometimes) <br />
# profiling. The Makefile generated by setup will assign the generic token <br />
# (ex. FFLAGS) to the proper set of flags (ex. FFLAGS_OPT).<br />
#----------------------------------------------------------------------------<br />
<br />
FFLAGS_OPT = -c -r8 -i4 -O3 -xSSE4.2<br />
FFLAGS_DEBUG = -c -g -r8 -i4 -O0<br />
FFLAGS_TEST = -c -r8 -i4<br />
<br />
LIB_HDF5 = -L${HDF5_PATH}/lib -lhdf5 -L${SCINET_ZLIB_LIB} -lz -lgpfs<br />
<br />
# if we are using HDF5, we need to specify the path to the include files<br />
CFLAGS_HDF5 = -I${HDF5_PATH}/include<br />
<br />
CFLAGS_OPT = -c -O3 -xSSE4.2<br />
CFLAGS_TEST = -c -O2 <br />
CFLAGS_DEBUG = -c -g <br />
<br />
MDEFS = <br />
<br />
.SUFFIXES: .o .c .f .F .h .fh .F90 .f90<br />
<br />
#----------------------------------------------------------------------------<br />
# Linker flags<br />
#<br />
# There is a seperate version of the linker flags for each of the _OPT, <br />
# _DEBUG, and _TEST cases.<br />
#----------------------------------------------------------------------------<br />
<br />
LFLAGS_OPT = -o<br />
LFLAGS_TEST = -o<br />
LFLAGS_DEBUG = -g -o<br />
<br />
MACHOBJ = <br />
<br />
<br />
MV = mv -f<br />
AR = ar -r<br />
RM = rm -f<br />
CD = cd<br />
RL = ranlib<br />
ECHO = echo<br />
</source><br />
<br />
<br />
-- [[User:Ljdursi|ljdursi]] 22:11, 13 August 2009 (UTC)<br />
<br />
==Aeronautics==<br />
<br />
==Chemistry==<br />
<br />
===CPMD===<br />
<br />
Please refer to the [[Cpmd | CPMD]] page.<br />
<br />
===NWChem===<br />
<br />
Please refer to the [[Nwchem | NWChem]] page.<br />
<br />
===GAMESS (US)===<br />
<br />
Please refer to the [[gamess|GAMESS (US)]] page.<br />
<br />
User supplied content below.<br />
<br />
====Tips from the Fekl Lab====<br />
<br />
Through trial and error, we have found a few useful things that we would like to share:<br />
<br />
1. Two very useful, open-source programs for visualization of output files from GAMESS(US) and for generation of input files are [http://www.scl.ameslab.gov/MacMolPlt/ MacMolPlt]and [http://avogadro.openmolecules.net/wiki/Main_Page Avogadro]. The are available for UNIX/LINUX, Windows and Mac based machines, HOWEVER: any input files that we have generated with these programs on a Windows-based machine do not run on Mac based machines. We don't know why.<br />
<br />
2. [http://winscp.net/eng/index.php WinSCP] is a very useful tool that has a graphical user interface for moving files from a local machine to SCINET and vice versa. It also has text editing capabilities.<br />
<br />
3. The [https://bse.pnl.gov/bse/portal ESML Basis Set Exchange] is an excellent source for custom basis set or effective core potential parameters. Make sure that you specify "Gamess-US" in the format drop-down box.<br />
<br />
4. The commercial program [http://www.chemcraftprog.com/ ChemCraft] is a highly useful visualization program that has the ability to edit molecules in a very similar fashion to GaussView. It can also be customized to build GAMESS(US) input files.<br />
<br />
====Anatomy of a GAMESS(US) Input File with Basis Set Info in an External File====<br />
<br />
$CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE DFTTYP=M06-L MAXIT=199 MULT=1 NOSYM=1<br />
ECP=READ $END<br />
$SYSTEM TIMLIM=525600 MWORDS=1750 PARALL=.TRUE. $END<br />
$BASIS GBASIS=CUSTOMNI EXTFIL=.t. $END<br />
$SCF DIRSCF=.TRUE. FDIFF=.f. $END<br />
$STATPT OPTTOL=0.0001 NSTEP=500 HSSEND=.t. $END<br />
$DATA<br />
Mo_BDT3<br />
C1<br />
MOLYBDENUM 42.0 5.7556500000 4.4039600000 16.5808400000<br />
SULFUR 16.0 7.4169700000 3.1956300000 15.2089300000<br />
SULFUR 16.0 4.0966800000 3.2258300000 15.1761100000<br />
SULFUR 16.0 3.9677300000 4.4940500000 18.3266100000<br />
SULFUR 16.0 7.1776900000 3.5815000000 18.4485200000<br />
SULFUR 16.0 4.3776600000 6.2447400000 15.6786900000<br />
SULFUR 16.0 7.5478700000 6.0679800000 16.2223700000<br />
CARBON 6.0 6.4716900000 2.1004800000 14.1902300000<br />
CARBON 6.0 5.0690300000 2.1781400000 14.1080700000<br />
CARBON 6.0 4.8421800000 4.2701300000 19.8855500000<br />
CARBON 6.0 6.1969000000 3.9249600000 19.9397400000<br />
CARBON 6.0 6.8280600000 3.7834200000 21.1913200000<br />
CARBON 6.0 5.7697600000 7.6933500000 17.4241800000<br />
CARBON 6.0 7.2043100000 7.9413600000 17.8281100000<br />
CARBON 6.0 5.5051400000 7.0409700000 14.5903800000<br />
CARBON 6.0 6.8905200000 6.9194700000 14.7626200000<br />
CARBON 6.0 7.7396400000 7.5379800000 13.8285700000<br />
HYDROGEN 1.0 8.8190700000 7.4520600000 13.9252200000<br />
CARBON 6.0 7.2169400000 8.2960300000 12.7704100000<br />
HYDROGEN 1.0 7.8667000000 8.7825100000 12.0575600000<br />
CARBON 6.0 5.8260300000 8.4502300000 12.6467800000<br />
HYDROGEN 1.0 5.4143000000 9.0544300000 11.8493100000<br />
CARBON 6.0 4.9881500000 7.8192300000 13.5528400000<br />
HYDROGEN 1.0 3.9090500000 7.9420000000 13.4583700000<br />
CARBON 6.0 7.1538500000 1.1569600000 13.4143900000<br />
CARBON 6.0 4.4018100000 1.3603900000 13.1919900000<br />
CARBON 6.0 6.4791600000 0.3185500000 12.5353300000<br />
CARBON 6.0 5.0837400000 0.4369500000 12.4084900000<br />
HYDROGEN 1.0 7.0116000000 -0.4099400000 11.9434600000<br />
HYDROGEN 1.0 8.2399000000 1.0702400000 13.4937600000<br />
HYDROGEN 1.0 3.3185600000 1.4368700000 13.0953100000<br />
HYDROGEN 1.0 4.5549800000 -0.1997300000 11.7165200000<br />
CARBON 6.0 6.1105700000 3.9639000000 22.3866100000<br />
CARBON 6.0 4.1216300000 4.4424400000 21.1020100000<br />
HYDROGEN 1.0 7.8732900000 3.5217100000 21.2520500000<br />
CARBON 6.0 4.7606000000 4.2868500000 22.3363800000<br />
HYDROGEN 1.0 6.6064200000 3.8406000000 23.3428500000<br />
HYDROGEN 1.0 4.2065000000 4.4170700000 23.2667100000<br />
HYDROGEN 1.0 3.0674000000 4.6893500000 21.0889000000<br />
HYDROGEN 1.0 7.4249200000 7.7545300000 18.8583200000<br />
HYDROGEN 1.0 7.6651700000 8.9049700000 17.7652100000<br />
HYDROGEN 1.0 5.3324000000 8.6487800000 17.2222700000<br />
HYDROGEN 1.0 5.5015000000 7.1039000000 18.2759400000<br />
$END<br />
$ECP<br />
MO-ECP GEN 28 3<br />
5 ----- f potential -----<br />
-0.0469492 0 537.9667807 <br />
-20.2080084 1 147.8982938 <br />
-106.2116302 2 45.7358898 <br />
-41.8107368 2 13.2911467 <br />
-4.2054103 2 4.7059961 <br />
3 ----- s-f potential -----<br />
2.8063717 0 110.2991760 <br />
44.5162012 1 23.2014645 <br />
82.7785227 2 5.3530131 <br />
4 ----- p-f potential -----<br />
4.9420876 0 63.2901397 <br />
25.8604976 1 23.3315302 <br />
132.4708742 2 24.6759423 <br />
57.3149794 2 4.6493040 <br />
5 ----- d-f potential -----<br />
3.0054591 0 104.4839977 <br />
26.3637851 1 66.2307245 <br />
183.3849199 2 39.1283176 <br />
98.4453068 2 13.1164437 <br />
22.4901377 2 3.6280263 <br />
S NONE<br />
S NONE<br />
S NONE<br />
S NONE<br />
S NONE<br />
S NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
C NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
$END<br />
<br />
=====The Input Deck=====<br />
<br />
Below is the input deck. It is where you tell GAMESS(US) what job type to execute and where all you individual parameters are entered for your specific job type. The example input deck below is for a geometry optimization and frequency calculation. This input deck is equivalent to the Gaussian job with "opt" and "freq" in the route section.<br />
<br />
$CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE DFTTYP=M06-L MAXIT=199 MULT=1 NOSYM=1<br />
ECP=READ $END<br />
$SYSTEM TIMLIM=2850 MWORDS=1750 MEMDDI=20 PARALL=.TRUE. $END<br />
$BASIS GBASIS=CUSTOMNI EXTFIL=.t. $END<br />
$SCF DIRSCF=.TRUE. FDIFF=.f. $END<br />
$STATPT OPTTOL=0.0001 NSTEP=500 HSSEND=.t. $END<br />
$DATA<br />
<br />
An important thing to note is the spacing. In the input deck, there must be 1 space at the beginning of each line of the input deck. If not, the job will fail. Most builders will insert this space anyway, but it helps to double check.<br />
<br />
The end of the input deck is marked by the "$DATA" line.<br />
<br />
=====Job Title Line=====<br />
<br />
The next line of the file is the job title. It can be anthing you wish, however, we have found that to be on the safe side, we avoide using symbols or spaces<br />
<br />
Mo_BDT3<br />
<br />
=====Symmetry Point Group=====<br />
<br />
The next line of the file is the symmetry point group of your molecule. Note that there is no leading space before the point group.<br />
<br />
C1<br />
<br />
=====Coordinates=====<br />
<br />
The next block of text is set aside for the coordinates of the molecule. This can be in internal (or z-matrix) format or cartesian coordinates. Note that there is no leading space before the coordinates. One may use the chemical symbol or the full name of each atom in the molecule. Note that the end of the coordinates is signified by an "$END", which MUST have one space preceding it. The coordinates below do NOT have any basis set information inserted. It is possible to insert basis set information directly into the input file. This is accomplished by obtaining the desired basis set parameters from the EMSL and then inserting them below each relevant atom. An example input file with inserted basis set information will be shown later.<br />
<br />
MOLYBDENUM 42.0 5.7556500000 4.4039600000 16.5808400000<br />
SULFUR 16.0 7.4169700000 3.1956300000 15.2089300000<br />
SULFUR 16.0 4.0966800000 3.2258300000 15.1761100000<br />
SULFUR 16.0 3.9677300000 4.4940500000 18.3266100000<br />
SULFUR 16.0 7.1776900000 3.5815000000 18.4485200000<br />
SULFUR 16.0 4.3776600000 6.2447400000 15.6786900000<br />
SULFUR 16.0 7.5478700000 6.0679800000 16.2223700000<br />
CARBON 6.0 6.4716900000 2.1004800000 14.1902300000<br />
CARBON 6.0 5.0690300000 2.1781400000 14.1080700000<br />
CARBON 6.0 4.8421800000 4.2701300000 19.8855500000<br />
CARBON 6.0 6.1969000000 3.9249600000 19.9397400000<br />
CARBON 6.0 6.8280600000 3.7834200000 21.1913200000<br />
CARBON 6.0 5.7697600000 7.6933500000 17.4241800000<br />
CARBON 6.0 7.2043100000 7.9413600000 17.8281100000<br />
CARBON 6.0 5.5051400000 7.0409700000 14.5903800000<br />
CARBON 6.0 6.8905200000 6.9194700000 14.7626200000<br />
CARBON 6.0 7.7396400000 7.5379800000 13.8285700000<br />
HYDROGEN 1.0 8.8190700000 7.4520600000 13.9252200000<br />
CARBON 6.0 7.2169400000 8.2960300000 12.7704100000<br />
HYDROGEN 1.0 7.8667000000 8.7825100000 12.0575600000<br />
CARBON 6.0 5.8260300000 8.4502300000 12.6467800000<br />
HYDROGEN 1.0 5.4143000000 9.0544300000 11.8493100000<br />
CARBON 6.0 4.9881500000 7.8192300000 13.5528400000<br />
HYDROGEN 1.0 3.9090500000 7.9420000000 13.4583700000<br />
CARBON 6.0 7.1538500000 1.1569600000 13.4143900000<br />
CARBON 6.0 4.4018100000 1.3603900000 13.1919900000<br />
CARBON 6.0 6.4791600000 0.3185500000 12.5353300000<br />
CARBON 6.0 5.0837400000 0.4369500000 12.4084900000<br />
HYDROGEN 1.0 7.0116000000 -0.4099400000 11.9434600000<br />
HYDROGEN 1.0 8.2399000000 1.0702400000 13.4937600000<br />
HYDROGEN 1.0 3.3185600000 1.4368700000 13.0953100000<br />
HYDROGEN 1.0 4.5549800000 -0.1997300000 11.7165200000<br />
CARBON 6.0 6.1105700000 3.9639000000 22.3866100000<br />
CARBON 6.0 4.1216300000 4.4424400000 21.1020100000<br />
HYDROGEN 1.0 7.8732900000 3.5217100000 21.2520500000<br />
CARBON 6.0 4.7606000000 4.2868500000 22.3363800000<br />
HYDROGEN 1.0 6.6064200000 3.8406000000 23.3428500000<br />
HYDROGEN 1.0 4.2065000000 4.4170700000 23.2667100000<br />
HYDROGEN 1.0 3.0674000000 4.6893500000 21.0889000000<br />
HYDROGEN 1.0 7.4249200000 7.7545300000 18.8583200000<br />
HYDROGEN 1.0 7.6651700000 8.9049700000 17.7652100000<br />
HYDROGEN 1.0 5.3324000000 8.6487800000 17.2222700000<br />
HYDROGEN 1.0 5.5015000000 7.1039000000 18.2759400000<br />
$END<br />
<br />
=====Effective Core Potential Data=====<br />
<br />
The effective core potential (ECP) data is entered after the coordinates. It starts with "$ECP", which must be preceded with a space. The atoms of the molecule are listed in the same order as in the coordinates section and the parameters for the ECP are listed after each atom. Note that for any atom that does NOT have an ECP, one must enter "ECP-NONE" or "NONE" after each atom without an ECP.<br />
<br />
$ECP<br />
MO-ECP GEN 28 3<br />
5 ----- f potential -----<br />
-0.0469492 0 537.9667807 <br />
-20.2080084 1 147.8982938 <br />
-106.2116302 2 45.7358898 <br />
-41.8107368 2 13.2911467 <br />
-4.2054103 2 4.7059961 <br />
3 ----- s-f potential -----<br />
2.8063717 0 110.2991760 <br />
44.5162012 1 23.2014645 <br />
82.7785227 2 5.3530131 <br />
4 ----- p-f potential -----<br />
4.9420876 0 63.2901397 <br />
25.8604976 1 23.3315302 <br />
132.4708742 2 24.6759423 <br />
57.3149794 2 4.6493040 <br />
5 ----- d-f potential -----<br />
3.0054591 0 104.4839977 <br />
26.3637851 1 66.2307245 <br />
183.3849199 2 39.1283176 <br />
98.4453068 2 13.1164437 <br />
22.4901377 2 3.6280263 <br />
S NONE<br />
S NONE<br />
S NONE<br />
S NONE<br />
S NONE<br />
S NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
C NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
C NONE<br />
C NONE<br />
H NONE<br />
C NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
H NONE<br />
$END<br />
<br />
-- [[User:M.Zimmer-De Iuliis|mzd]] 16 November 2009<br />
<br />
====Using an External File to Define Basis Set in GAMESS(US)====<br />
<br />
Since GAMESS(US) has a limited number of built-in ECPs and basis sets, one may want to make GAMESS(US) read an external file that contains the basis set information ECP data using the "EXTFIL" keyword in the $GBASIS command line of the input file. For many metal containing compounds, it is very convenient and time saving to use an effective core potential (ECP) for the core metal electrons, as they are usually not important to the reactivity of the complex or the geometry around the metal. In addition, to make GAMESS(US) use this external file, one must copy the "rungms" file and modify it accordingly. The following is a list of instructions with commands that will work from a terminal. One could also use WinSCP to do all of this with a GUI rather than a TUI. <br />
<br />
=====Modifiying rungms to Use Custom Basis Set File=====<br />
1. Copy "rungms" from /scinet/gpc/Applications/gamess to one's own /scratch/$USER/ directory:<br />
cp /scinet/gpc/Applications/gamess/rungms /scratch/$USER/<br />
<br />
2. Change to the scratch directory and check to see if "rungms" has copied successfully.<br />
cd /scratch/$USER<br />
ls<br />
<br />
3. Edit line 147 of the script. <br />
vi rungms<br />
Move the cursor down to line 147 using the arrow keys. It should say "setenv EXTBAS /dev/null". Using the arrow keys, move the cursor to the first "/" and then hit "i" to insert text. Put the path to your external basis file here. For example, /scratch/$USER/basisset. Then hit "escape". To save the changes and exit vi, type ":" and you should see a colon appear at the bottom of the window. Type "wq" (which should appear at the bottom of the window next to the colon) and then hit enter. Now you are done with vi.<br />
<br />
=====Creating a Custom Basis Set File=====<br />
1. To create a custom basis set file, you need create a new text document. Our group's common practice is to comment out the first line of this file by inserting an exclamation mark (!) followed by noting the specific basis sets and ECPs that are going to be used for each of the atoms. Let us use the molecule Mo(CO)6, Molybdenum hexacarbonyl, as an example. Below is the first line of the the external file, which we will call "CUSTOMMO" (NOTE: you can use any name for the external file that suits you, as long as it has no spaces and is 8 characters or less).<br />
<br />
! 6-31G on C and O and LANL2D2 ECP on Mo<br />
<br />
2. The next step is to visit the [https://bse.pnl.gov/bse/portal EMSL Basis Set exchange] and select C and O from the periodic table. Then, on the left of the page, select "6-31G" as the basis set. Finally, make sure the output is in GAMESS(US) format using the drop-down menu and then click "get basis set".<br />
<br />
[[File:C_O_6_31G_basisset.JPG|centre]]<br />
<br />
3. A new window should appear with text in it. For our example case, the text looks like this:<br />
<br />
! 6-31G EMSL Basis Set Exchange Library 10/13/09 11:12 AM<br />
! Elements References<br />
! -------- ----------<br />
! H - He: W.J. Hehre, R. Ditchfield and J.A. Pople, J. Chem. Phys. 56,<br />
! Li - Ne: 2257 (1972). Note: Li and B come from J.D. Dill and J.A.<br />
! Pople, J. Chem. Phys. 62, 2921 (1975).<br />
! Na - Ar: M.M. Francl, W.J. Petro, W.J. Hehre, J.S. Binkley, M.S. Gordon,<br />
! D.J. DeFrees and J.A. Pople, J. Chem. Phys. 77, 3654 (1982)<br />
! K - Zn: V. Rassolov, J.A. Pople, M. Ratner and T.L. Windus, J. Chem. Phys.<br />
! 109, 1223 (1998)<br />
! Note: He and Ne are unpublished basis sets taken from the Gaussian<br />
! program<br />
! <br />
$DATA<br /><br />
CARBON<br />
S 6<br />
1 3047.5249000 0.0018347 <br />
2 457.3695100 0.0140373 <br />
3 103.9486900 0.0688426 <br />
4 29.2101550 0.2321844 <br />
5 9.2866630 0.4679413 <br />
6 3.1639270 0.3623120 <br />
L 3<br />
1 7.8682724 -0.1193324 0.0689991 <br />
2 1.8812885 -0.1608542 0.3164240 <br />
3 0.5442493 1.1434564 0.7443083 <br />
L 1<br />
1 0.1687144 1.0000000 1.0000000<br /> <br />
OXYGEN<br />
S 6<br />
1 5484.6717000 0.0018311 <br />
2 825.2349500 0.0139501 <br />
3 188.0469600 0.0684451 <br />
4 52.9645000 0.2327143 <br />
5 16.8975700 0.4701930 <br />
6 5.7996353 0.3585209 <br />
L 3<br />
1 15.5396160 -0.1107775 0.0708743 <br />
2 3.5999336 -0.1480263 0.3397528 <br />
3 1.0137618 1.1307670 0.7271586 <br />
L 1<br />
1 0.2700058 1.0000000 1.0000000 <br />
$END<br />
<br />
3. Now, copy and paste the text between the $DATA and $END headings onto our external text file, CUSTOMMO. We also need to change the change the name of each element to the corresponding symbol in the periodic table. Finally, we need to add the name of the external file next to the element symbol, separated by one space. Note that there should be a blank line separating the basis set information and the first, commented-out line (The line starting with the '!'). The CUSTOMMO should look like this:<br />
<br />
! 6-31G on C and O and LANL2D2 ECP on Mo<br /><br />
C CUSTOMMO<br />
S 6<br />
1 3047.5249000 0.0018347 <br />
2 457.3695100 0.0140373 <br />
3 103.9486900 0.0688426 <br />
4 29.2101550 0.2321844 <br />
5 9.2866630 0.4679413 <br />
6 3.1639270 0.3623120 <br />
L 3<br />
1 7.8682724 -0.1193324 0.0689991 <br />
2 1.8812885 -0.1608542 0.3164240 <br />
3 0.5442493 1.1434564 0.7443083 <br />
L 1<br />
1 0.1687144 1.0000000 1.0000000<br /> <br />
O CUSTOMMO<br />
S 6<br />
1 5484.6717000 0.0018311 <br />
2 825.2349500 0.0139501 <br />
3 188.0469600 0.0684451 <br />
4 52.9645000 0.2327143 <br />
5 16.8975700 0.4701930 <br />
6 5.7996353 0.3585209 <br />
L 3<br />
1 15.5396160 -0.1107775 0.0708743 <br />
2 3.5999336 -0.1480263 0.3397528 <br />
3 1.0137618 1.1307670 0.7271586 <br />
L 1<br />
1 0.2700058 1.0000000 1.0000000 <br />
<br />
4. Repeat Step 3 above but choose Mo and select the LANL2DZ ECP instead. A new window will pop up with the basis set information as well as the ECP data we need, since we specified the LANL2DZ '''ECP'''. The ECP data is not inserted into the external file, rather it is placed into the input file itself (More on this later). <br />
<br />
[[File:Mo_LANL2DZ_basisset.JPG|centre]]<br />
<br />
5. After copying the molybdenum basis set information, your fiished external basis set file should look like this:<br />
! 6-31G on C and O and LANL2D2 ECP on Mo<br /><br />
C CUSTOMMO<br />
S 6<br />
1 3047.5249000 0.0018347 <br />
2 457.3695100 0.0140373 <br />
3 103.9486900 0.0688426 <br />
4 29.2101550 0.2321844 <br />
5 9.2866630 0.4679413 <br />
6 3.1639270 0.3623120 <br />
L 3<br />
1 7.8682724 -0.1193324 0.0689991 <br />
2 1.8812885 -0.1608542 0.3164240 <br />
3 0.5442493 1.1434564 0.7443083 <br />
L 1<br />
1 0.1687144 1.0000000 1.0000000<br /> <br />
O CUSTOMMO<br />
S 6<br />
1 5484.6717000 0.0018311 <br />
2 825.2349500 0.0139501 <br />
3 188.0469600 0.0684451 <br />
4 52.9645000 0.2327143 <br />
5 16.8975700 0.4701930 <br />
6 5.7996353 0.3585209 <br />
L 3<br />
1 15.5396160 -0.1107775 0.0708743 <br />
2 3.5999336 -0.1480263 0.3397528 <br />
3 1.0137618 1.1307670 0.7271586 <br />
L 1<br />
1 0.2700058 1.0000000 1.0000000<br /> <br />
Mo CUSTOMO<br />
S 3<br />
1 2.3610000 -0.9121760 <br />
2 1.3090000 1.1477453 <br />
3 0.4500000 0.6097109 <br />
S 4<br />
1 2.3610000 0.8139259 <br />
2 1.3090000 -1.1360084 <br />
3 0.4500000 -1.1611592 <br />
4 0.1681000 1.0064786 <br />
S 1<br />
1 0.0423000 1.0000000 <br />
P 3<br />
1 4.8950000 -0.0908258 <br />
2 1.0440000 0.7042899 <br />
3 0.3877000 0.3973179 <br />
P 2<br />
1 0.4995000 -0.1081945 <br />
2 0.0780000 1.0368093 <br />
P 1<br />
1 0.0247000 1.0000000 <br />
D 3<br />
1 2.9930000 0.0527063 <br />
2 1.0630000 0.5003907 <br />
3 0.3721000 0.5794024 <br />
D 1<br />
1 0.1178000 1.0000000<br />
<br />
-- [[User:M.Zimmer-De Iuliis|mzd]] 21 September 2010<br />
<br />
====A Modified BASH Script for Runnning GAMESS(US)====<br />
Below please find the bash script that we use to run GAMESS(US) on a single node with 8 processors. <br />
<br />
One quirk of GAMESS(US) is that it will NOT write over old or failed jobs that have the same name as the input file you are submitting. For example: my input file name is "mo_opt.inp" and I submit this job to the queue. However, it comes back seconds later with an error. The log file says that I have typed an incorrect keyword, and lo and behold, I have a comma where it shouldn't be. Such typos can be common. If you simply try to re-submit, GAMESS(US) will fail again, because it has written a .log file and some other files to the /scratch/user/gamess-scratch/ directory. These files must all be deleted before you re-submit your fixed input file.<br />
<br />
This script takes care of this annoying problem by deleting failed jobs with the same file name for you.<br />
<br />
Here it is:<br />
<br />
#!/bin/bash<br />
#PBS -l nodes=1:ppn=8,walltime=48:00:00<br />
<br />
## To submit type: qsub x.sh<br />
<br />
# If not an interactive job (i.e. -I), then cd into the directory where<br />
# I typed qsub.<br />
if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then<br />
if [ -n "$PBS_O_WORKDIR" ]; then<br />
cd $PBS_O_WORKDIR<br />
fi<br />
fi<br />
<br />
# the input file is typically named something like "gamesjob.inp"<br />
# so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"<br />
<br />
find /scratch/user/gamess-scratch -type f -name ${NAME:-safety_net}\* -exec /bin/rm {} \;<br />
<br />
# loading modules should be done in the submission script.<br />
# It is not recommended by SciNet staff to put this command<br />
# in the .bashrc file.<br />
module load gamess<br />
<br />
# run the program<br />
<br />
/scratch/user/rungms $NAME 00 8 8 >& $NAME.log<br />
<br />
====A Script to Add the $VIB Group for Hessian Restarts in GAMESS(US)====<br />
<br />
Sometimes, a optimization + vibrational analysis or just a plain vibrational analysis must be restarted. This can be because the two day time limit has been exceeded or perhaps there was an error during calculation. In any case, when this happens, the job must be restarted. In GAMESS(US), you can restart a vibrational analysis from a previous one and it will utilize the frequencies that were already computed in the failed run.<br />
<br />
For example, if one submits the input file "job_name.inp" and it fails before it has finished, then one must utilize the file "job_name.rst", which contains data that is required to restart the calculation. This file is located in the /scratch/user/gamess-scratch directory. Data from the "job_name.rst" file must be appended at the end of the new input file (after the coordinates and ECP section if it is present) to restart the calculation, letus call it "job_name_restart.inp"<br />
<br />
A shortened version of the "job_name.rst" file looks like this:<br />
<br />
ENERGY/GRADIENT/DIPOLE RESTART DATA FOR RUNTYP=HESSIAN<br />
job_name <br />
$VIB <br />
IVIB= 0 IATOM= 0 ICOORD= 0 E= -3717.1435124522<br />
-5.165258381E-04 1.584665821E-02-1.206270555E-02-2.241461728E-03 3.176050715E-03<br />
-5.706738823E-04 2.502034151E-03 5.130112290E-04-2.716945939E-03 1.357008279E-03<br />
-1.059915305E-03 1.693526456E-03-2.957638907E-04-5.994938737E-04 9.684054361E-04<br />
.<br />
.<br />
.<br />
.<br />
<br />
<br />
The text eventually ends with one blank line. The $VIB heading and all of the text after $VIB must be appended to the end of file "job_name_restart.inp" and then " $END" must be inserted at the very end of the file.<br />
<br />
One could do this, one could cut cut and paste in a text editor, but we have written a small script that will do this automatically. We call it ".vib.sh" but you can call it whatever you like. Here it is:<br />
<br />
#!/bin/bash<br />
# script to add vibrational data for a hessian restart<br />
<br />
awk '/\$VIB/{p=1}p;END{print " $END"}' /scratch/user/gamess-scratch/$NAME1.rst >> $NAME2.inp<br />
<br />
To use it, simply copy it into a new text file with the extension ".sh" and make it executable. Also, you will need to edit the location of the "/scratch/user/gamess-scratch/ directory to match your user name. The two variables in the script, NAME1 and NAME2, represent the name of your ".rst" file and your new ".inp" file, respectively. In the example above, NAME1=job_name (that is, the same name as the .rst file that contains the $VIB data and that was created in the /gamess-scrsatch/ directory) and NAME2=job_name_restart (that is, the name of the new input file that you have prepared and want to copy the $VIB data into).<br />
<br />
To run it on a gpc node without submitting it to the job queue, type:<br />
<br />
NAME1=job_name NAME2=job_name_restart ./vib.sh<br />
<br />
To run it in the queue, type:<br />
<br />
qsub vib.sh -v NAME1=job_name,NAME2=job_name_restart <br />
<br />
-special thanks to Ramses for help with this<br />
<br />
-- [[User:M.Zimmer-De Iuliis|mzd]] 30 September 2010<br />
<br />
====Most Commonly Used Headers in The Fekl Lab====<br />
<br />
After about a year of using GAMESS(US), we have found that we are most often doing optimizations, frequency analyses, transition state searches and IRC calculations using DFT methods. Here are the input decks thatwe found have worked well for inorganic and organometallic compounds.<br />
<br />
=====Optimization Plus Frequency (for a neutral, singlet)=====<br />
<br />
$CONTRL SCFTYP=RHF RUNTYP=OPTIMIZE DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1<br />
ECP=READ $END<br />
$SYSTEM TIMLIM=2800 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END<br />
$SCF DIRSCF=.TRUE. FDIFF=.f. DIIS=.T. SOSCF=.F. DAMP=.T. $END<br />
$STATPT OPTTOL=0.00001 NSTEP=500 HSSEND=.t. $END<br />
$FORCE TEMP=298.15 PURIFY=.t. PROJCT=.t. $END<br />
$DATA<br />
<br />
=====Frequency Only (for a neutral, singlet)=====<br />
<br />
$CONTRL SCFTYP=RHF RUNTYP=HESSIAN DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1<br />
ECP=READ $END<br />
$SYSTEM TIMLIM=2800 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END<br />
$SCF DIRSCF=.TRUE. FDIFF=.f. DIIS=.T. SOSCF=.F. DAMP=.T. $END<br />
$FORCE METHOD=SEMINUM VIBANL=.TRUE. PROJCT=.T. PURIFY=.T. $END<br />
$DATA<br />
<br />
=====Transition State Search (for a neutral, singlet)=====<br />
<br />
$CONTRL SCFTYP=RHF RUNTYP=SADPOINT DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1<br />
ECP=READ $END<br />
$SYSTEM TIMLIM=2850 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END<br />
$SCF DIRSCF=.TRUE. FDIFF=.f. DIIS=.T. SOSCF=.F. $END<br />
$STATPT STSTEP=0.05 OPTTOL=0.00001 NSTEP=500 HESS=CALC HSSEND=.t. <br />
STPT=.FALSE. $END<br />
$FORCE METHOD=SEMINUM VIBANL=.TRUE. PURIFY=.T. PROJCT=.T. $END<br />
$DATA<br />
<br />
=====IRC (Intrinsic Reaction Coordinate following forward reaction) Calculation (for a neutral, singlet)=====<br />
<br />
$CONTRL SCFTYP=RHF RUNTYP=IRC DFTTYP=''FILL_IN_YOUR_PREFEENCE_HERE'' MAXIT=199 MULT=1 NOSYM=1<br />
ECP=READ $END<br />
$IRC OPTTOL=0.00001 STRIDE=0.05 NPOINT=5000 SADDLE=.TRUE. FORWRD=.F.<br />
$END<br />
$SYSTEM TIMLIM=2850 MWORDS=20 MEMDDI=50 PARALL=.TRUE. $END<br />
$SCF DIRSCF=.TRUE. FDIFF=.f. $END<br />
$FORCE TEMP=298.15 PURIFY=.t. PROJCT=.t. $END<br />
$DATA<br />
<br />
-- [[User:M.Zimmer-De Iuliis|mzd]] 21 September 2010<br />
<br />
====How to Run an IRC Calculation Using GAMESS(US)====<br />
<br />
An IRC or Intrinsic Reaction Coordinate calculation follows the imaginary mode of the vibrational analysis of a transition state calculation. In GAMESS(US), you can choose to follow the forward (towards the products) or backward (toward the reactants) direction. As shown above in the IRC header that we use, the direction of the IRC calculation is controlled by the "FORWRD" key word. Using "FORWRD=.T." means that the IRC is following the forward direction, while using "FORWRD=.F." means that the IRC calculation is following the backward direction.<br />
<br />
Let us say we want to perform an IRC. In order to perform an IRC calculation, you must first perform a vibrational analysis of you molecule and check to ensure there is only 1 negative frequency. If that is the case, then the vibrational analysis completed successfully and there will be a file, let us call it "job_name.dat" in the "/users/$USER/gamess-scratch/" directory (where $USER is your user name) with the extension ".dat". In this file is data that is required for the IRC input file.<br />
<br />
To prepare your IRC input file, prepare an input file using the coordinates of the optimized structure of the transition state. This can be from ChemCraft or Avogadro or MacMolPlt - what ever you prefer to use. Then copy and paste the IRC header above or use your own parameters. Call it whatever you want, as long as it has an ".inp" extension. Let us call in "irc_job.inp". <br />
<br />
For example, the "STRIDE" value determines the "size" of the steps between each point on the IRC graph. If you increase the value of the stride, say from 0.05 to 0.1, then the steps in between each point become larger and you will approach the minimum faster (this will give you fewer data points should you chose to plot the IRC data). Decreasing the stride value, say from 0.05 to 0.01 will make the steps in between each point become smaller and you may not reach the minimum of the reaction coordinate in the alloted time period.<br />
<br />
You should now have an input file with an IRC header, the coordinates of the transition state and basis set and ECP information called "irc_job.inp".<br />
<br />
Now you need to use the "job_name.dat" file in the "/users/$USER/gamess-scratch/" In this file are a number of blocks of data that are sandwiched between a line that contains only " $HESS" and a line that contains only " $END". What you need is the LAST of these blocks of text and it has to be copied and pasted directly below the last entry of your input file.<br />
<br />
This can be difficult and time consuming, as the .dat files can be very large (sometimes over 150 MB) and cumbersome to navigate through. However, we have written a script, similar to the .vib.sh script, that can help you out with this. Basically, this script does all the copying and pasting for you. <br />
<br />
Here it is:<br />
#!/bin/bash<br />
# script to add hessian data for an IRC calculation<br />
<br />
awk '/\$HESS/{arr="";f=1} f {arr=(arr)?arr ORS $0:$0} /\$END/{f=0} END {print arr}' /scratch/$USER/gamess-scratch/$DAT.dat >> $IN.inp<br />
<br />
To use it, simply copy it into a new text file with the name "irc.sh" and make it executable. Also, you will need to edit the location of the "/scratch/user/gamess-scratch/ directory to match your user name. The two variables in the script, $DAT and $IN, represent the name of your ".dat" file and your new ".inp" file, respectively. Using our current example, $DAT=job_name and In the example above, $IN=irc_job (that is, the same name as the .dat file that contains the $HESS data and that was created in the /gamess-scrsatch/ directory) and IN=irc_job (that is, the name of the new input file that you have prepared and want to copy the $HESS data into). <br />
<br />
To run it on a gpc node without submitting it to the job queue, type:<br />
<br />
DAT=job_name IN=irc_job ./irc.sh<br />
<br />
To run it in the queue, type:<br />
<br />
qsub irc.sh -v DAT=job_name,IN=irc_job <br />
<br />
-- [[User:M.Zimmer-De Iuliis|mzd]] 21 October 2010<br />
<br />
===Vienna Ab-initio Simulation Package (VASP)===<br />
Please refer to the [http://www.vasp.at VASP page].<br />
<br />
User-supplied content below.<br />
<br />
====Tips from the Polanyi Lab====<br />
Using VASP on SciNet<br />
<br />
Logon using SSH<br />
login.scinet.utoronto.ca<br />
<br />
then ssh to the TCS cluster<br />
ssh tcs01<br />
<br />
change directory to <br />
cd /scratch/imcnab/test/Si111 - or whatever other directory is convenient.<br />
<br />
VASP is contained in the directory imcnab/bin<br />
<br />
To submit a job, first edit (at least) the POSCAR file and other VASP<br />
input files as necessary.<br />
<br />
=====Input Files=====<br />
The minimum set of input files is:<br />
<br />
'''vasp.script''' - script file telling TCS to run a VASP job - must be edited to run in current working directory.<br />
<br />
'''POSCAR''' - specifies supercell geometry and "ionic" positions (i.e. atomic centres) and whether relaxation allowed. Ionic positions may be given in cartesion coordinates (x,y,z in A) or "absolute", which are fractions of the unit cell vectors. CONTCAR is always in absolute coords, so after the first run of any job, you'll find yourself running in absolute coords. VMD can be used to change these back to cartesian coordinates.<br />
<br />
'''INCAR''' - specifies parameters to run the job. INCAR is free format - can put input commands in ANY order.<br />
<br />
'''POTCAR''' - specifies the potentials to use for each atomic type. Must be in the same order as the atoms are first met in POSCAR<br />
<br />
'''KPOINTS''' - specifies the number and position of K-points to use in the calculation.<br />
<br />
Any change of name or directory needs to be edited into the job script. The job script name is "vasp.script".<br />
<br />
VASP attempts to read initial wavefunctions from WAVECAR, so if a job is run in steps, leaving the WAVECAR file on the working directory is an efficient way to start the next stage of the calculation<br />
<br />
VASP also writes CONTCAR which is of the same format as POSCAR, and can simply be renamed if it is to be used as the starting point for a new job.<br />
<br />
<br />
Submit the job to load-leveller with the command llsubmit ./vasp.script from the correct working directory.<br />
<br />
can check the status of a job with llq<br />
<br />
can cancel a job using llcancel tcs-fXXnYY.$PID where tcs number etc is shown by llq<br />
<br />
== <br />
INPUT FILES ==<br />
<br />
The minimum set of input files is:<br />
<br />
'''vasp.script''' - script file telling TCS to run a VASP job - must be edited to run in current working directory.<br />
<br />
'''POSCAR''' - specifies supercell geometry and "ionic" positions (i.e. atomic centres) and whether relaxation allowed. Ionic positions may be given in cartesion coordinates (x,y,z in A) or "absolute", which are fractions of the unit cell vectors. CONTCAR is always in absolute coords, so after the first run of any job, you'll find yourself running in absolute coords. VMD can be used to change these back to cartesian coordinates.<br />
<br />
'''INCAR''' - specifies parameters to run the job. INCAR is free format - can put input commands in ANY order.<br />
<br />
'''POTCAR''' - specifies the potentials to use for each atomic type. Must be in the same order as the atoms are first met in POSCAR<br />
<br />
'''KPOINTS''' - specifies the number and position of K-points to use in the calculation.<br />
<br />
Any change of name or directory needs to be edited into the job script.<br />
The job script name is "'''vasp.script'''".<br />
<br />
VASP attempts to read initial wavefunctions from WAVECAR, so if a job is<br />
run in steps, leaving the WAVECAR file on the working directory is an <br />
efficient way to start the next stage of the calculation<br />
<br />
VASP also writes CONTCAR which is of the same format as POSCAR, and can<br />
simply be renamed if it is to be used as the starting point for a new job.<br />
<br />
<br />
Submit the job to load-leveller with the command<br />
llsubmit ./vasp.script from the correct working directory.<br />
<br />
can check the status of a job with <br />
llq<br />
<br />
can cancel a job using<br />
llcancel tcs-fXXnYY.$PID where tcs number etc is shown by llq<br />
<br />
<br />
<br />
===== GENERAL NOTES =====<br />
<br />
MUCH faster to use ISPIN=1, no-spin (corresponds to RHF, rather than <br />
ISPIN=2 which corresponds to URHF). So far, I've not found a system where the atom positions differ, or where the calculated electronic energy differs by more than 1E-4, which is the convergence <br />
criteria set.<br />
<br />
MUCH faster to use real space LREAL = A, NSIM=4. <br />
<br />
So, ''always'' optimize in real space first, then re-optimize in reciprocal space. This does NOT guarantee, a one-step optimization in reciprocal space. May still need to progressively<br />
relax a large system.<br />
<br />
'''Relaxing a large system.'''<br />
If you attempt to relax a large system in one step, it will usually fail.<br />
<br />
The starting geometry is usually an unrelaxed molecule above an unrelaxed surface.<br />
The bottom plane of the surface will NEVER be relaxed, because this corresponds to the fixed boundary condition of REALITY. <br />
<br />
First, relax the molecule alone (assuming you have already found a good starting position from single point calcultions, place the molecule closer to the surface than you think it should be (say 0.9 VdW radii away).<br />
<br />
Then ALSO allow the top layer of the surface to relax.<br />
Then ALSO allow the second top layer of the surface to relax... etc... etc.<br />
<br />
If this DOESN'T WORK: Then relax X,Y and Z separately in iterations.<br />
Example. For the following problem, representing layers of the crystal going DOWN from the top (Z pointing to the top of the screen)<br />
<br />
Molecule<br /><br />
Layer 1<br /><br />
Layer 2<br /><br />
Layer 3<br /><br />
Layer 4<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
we can try the following relaxation schemes:<br /><br />
<br />
Successive relaxation, Layer by Layer:<br /><br />
(1) <br /><br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ fixed<br /><br />
Layer 2 XYZ fixed<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer.<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
(2)<br /> <br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ Relax<br /><br />
Layer 2 XYZ fixed<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer.<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
(3)<br /> <br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ Relax<br /><br />
Layer 2 XYZ Relax<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer.<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
etc. etc... if this works then you're fine. However, it can happen that even by Layer 2, you're running into real problems, and the ionic relaxation NEVER converges. In which case, I have found the following scheme (and variations thereof) useful:<br />
<br />
(1)<br /> <br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ fixed<br /><br />
Layer 2 XYZ fixed<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
(2) <br /><br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ Relax<br /><br />
Layer 2 XYZ fixed<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
(3) <br /><br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ Relax<br /><br />
Layer 2 XYZ Relax<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
IF (3) DOESN'T converge THEN TRY<br />
<br />
(2')<br /><br />
Molecule Z Relax, XY FIXED<br /><br />
Layer 1 Z Relax, XY FIXED<br /><br />
Layer 2 XYZ Relax<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
- you are allowing the top layers to move only UP or DOWN, while allowing the intermediate<br />
layer 2 to fully relax (actually, there is no way of telling VASP to move ALL atoms by the SAME deltaZ, but that appears to be the effect.<br />
Followed by<br />
<br />
(2")<br /><br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ Relax<br /><br />
Layer 2 XYZ Relax<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
If (2") doesn't work, you need to go back to the output of (2') and vary the cycle - perhaps something like:<br />
(2"')<br /><br />
Molecule XYZ Relax<br /><br />
Layer 1 XYZ Relax<br /><br />
Layer 2 XYZ fixed<br /><br />
Layer 3 XYZ fixed<br /><br />
Layer 4 XYZ fixed<br /><br />
Layer 5 - fixed layer<br /><br />
Layer 6 - Valence H's, fixed layer<br /><br />
<br />
then try (2") again.<br />
<br />
Repeat as necessary. This scheme does appear to work quite well for big unit cells. It can be very difficult to relax as many layers as necessary in a big unit cell.<br />
<br />
Experience on the One Per Corner Hole problem shows that it may be necessary to have a large number of UNRELAXED (i.e. BULK silicon) layers underneath the relaxed layers in order to get physically meaningful answers. This is because silicon is so elastic.<br />
<br />
===== Problems and solutions: =====<br />
<br />
If getting ZBRENT errors, try changing ALGO. Usually use ALGO = Fast, change to ALGO = Normal. With ALGO = Normal, NFREE now DOES correspond to degrees of freedom (maximum suggested setting is 20). Haven't found this terribly helpful.<br />
<br />
Many calculations seem to fail after 20 or 30 ionic steps. I suspect a memory leak.<br />
<br />
Sometimes the calculation appears to lose WAVECAR... this is not a disaster, just means a slight increase in start time as the first wavefunction is calculated.<br />
<br />
If calculation does not finish nicely, can force a WAVECAR generation by doing a purely electronic calculation (these are pretty fast).<br />
<br />
VASP is VERY slow at relaxing molecules at surfaces. This is because it doesn't know a molecule is a connected entity. It treats every atom independently. <br />
<br />
THEREFORE, MUCH MUCH faster to try molecular positions by hand first. <br />
Do some sample calculations at a few geometries to find a good starting point.<br />
<br />
ALSO, once you think you know where the molecule is to be placed, put it too close to the surface, and let it relax outwards... the forces close to the surface are repulsive, and much steeper, so relaxation is FASTER in this direction.<br />
<br />
=='''Climate Modelling'''==<br />
<br />
The Community Earth System Model (CESM) is a fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth's past, present, and future climate states.<br />
<br />
Development of a comprehensive CESM that accurately represents the principal components of the climate system and their couplings requires both wide intellectual participation and computing capabilities beyond those available to most U.S. institutions. The CESM, therefore, must include an improved framework for coupling existing and future component models developed at multiple institutions, to permit rapid exploration of alternate formulations. This framework must be amenable to components of varying complexity and at varying resolutions, in accordance with a balance of scientific needs and resource demands. In particular, the CESM must accommodate an active program of simulations and evaluations, using an evolving model to address scientific issues and problems of national and international policy interest.<br />
<br />
User guides and information on each version of the model can be found at the following links:<br />
<br />
CCSM3: http://www.cesm.ucar.edu/models/ccsm3.0/<br />
CCSM4: http://www.cesm.ucar.edu/models/ccsm4.0/<br />
CESM1: http://www.cesm.ucar.edu/models/cesm1.0/<br />
<br />
Please see:<br />
<br />
===[[Installing CCSM3]]===<br />
<br />
===[[Running CCSM3]]===<br />
<br />
===[[Installing CCSM4]]===<br />
<br />
===[[Running CCSM4]]===<br />
<br />
===[[Post Processing CCSM Output]]===<br />
<br />
===[[CCSM4/CESM1 TCS Simulation List]]===<br />
<br />
==Engineering==<br />
<br />
===ANSYS===<br />
<br />
Please refer to the [[Ansys]] page.<br />
<br />
==Medicine/Bio==<br />
<br />
==High Energy Physics==<br />
<br />
==Structural Biology==<br />
Molecular simulation of proteins, lipids, carbohydrates, and other biologically relevant molecules.<br />
===Molecular Dynamics (MD) simulation===<br />
====GROMACS====<br />
Please refer to the [[gromacs|GROMACS]] page<br />
====AMBER====<br />
Please refer to the [[amber|AMBER]] page<br />
====NAMD====<br />
'''Note: the instructions below are rather old, and should not be needed by most users. A version of namd is available on the GPC in the module <tt>namd</tt>. For current information see the [[Namd]] page'''<br />
<br />
NAMD is one of the better scaling MD packages out there. With sufficiently large systems, it is able to scale to hundreds or thousands of cores on Scinet. Below are details for compiling and running NAMD on Scinet.<br />
<br />
More information regarding performance and different compile options coming soon...<br />
<br />
=====Compiling NAMD for GPC=====<br />
Ensure the proper compiler/mpi modules are loaded.<br />
<source lang="bash"><br />
module load intel<br />
module load openmpi/1.3.3-intel-v11.0-ofed<br />
</source><br />
<br />
'''Compile Charm++ and NAMD'''<br />
<source lang="bash"><br />
#Unpack source files and get required support libraries<br />
tar -xzf NAMD_2.7b1_Source.tar.gz<br />
cd NAMD_2.7b1_Source<br />
tar -xf charm-6.1.tar<br />
wget http://www.ks.uiuc.edu/Research/namd/libraries/fftw-linux-x86_64.tar.gz<br />
wget http://www.ks.uiuc.edu/Research/namd/libraries/tcl-linux-x86_64.tar.gz<br />
tar -xzf fftw-linux-x86_64.tar.gz; mv linux-x86_64 fftw<br />
tar -xzf tcl-linux-x86_64.tar.gz; mv linux-x86_64 tcl<br />
#Compile Charm++<br />
cd charm-6.1<br />
./build charm++ mpi-linux-x86_64 icc --basedir /scinet/gpc/mpi/openmpi/1.3.3-intel-v11.0-ofed/ --no-shared -O -DCMK_OPTIMIZE=1<br />
cd ..<br />
#Compile NAMD. <br />
#Edit arch/Linux-x86_64-icc.arch and add "-lmpi" to the end of the CXXOPTS and COPTS line.<br />
#Make a builds directory if you want different versions of NAMD compiled at the same time.<br />
mkdir builds<br />
./config builds/Linux-x86_64-icc --charm-arch mpi-linux-x86_64-icc<br />
cd builds/Linux-x86_64-icc/<br />
make -j4 namd2 # Adjust value of j as desired to specify number of simultaneous make targets. <br />
</source><br />
--[[User:Cmadill|Cmadill]] 16:18, 27 August 2009 (UTC)<br />
<br />
=====Running Fortran=====<br />
On the development nodes, there is an old gcc. The associated libraries are not on the compute nodes. Ensure the line:<br />
<br />
module load gcc<br />
<br />
is in your .bashrc file.<br />
<br />
====LAMMPS====<br />
[[Image:StrongScalingLAMMPS.png|thumb|320px|right|Strong scaling test on GPC with OpenMPI and IntelMPI on Ethernet and InfiniBand]]<br />
[[Image:WeakScalingLAMMPS.png|thumb|320px|right|Weak scaling test on GPC with OpenMPI and IntelMPI on Ethernet and InfiniBand]]<br />
LAMMPS is a parallel MD code that can be found [http://lammps.sandia.gov/ here].<br />
<br />
'''Compiling LAMMPS on GPC'''<br />
<br />
First, make sure the ''intel/intel-v11.1.072'' and ''openmpi/1.4.1-intel-v11.0-ofed'' modules are loaded.<br />
<br />
To download LAMMPS into your directory, instructions can be found [http://lammps.sandia.gov/download.html here].<br />
<br />
In ''src/MAKE/Makefile.openmpi'', make the following changes:<br />
<br />
FFT_INC = -DFFT_NONE<br />
<br />
<nowiki># FFT_PATH =</nowiki> <br />
<br />
<nowiki># FFT_LIB =</nowiki><br />
<br />
Note that the FFT library is required for using PPPM from the KSPACE package.<br />
<br />
In the ''src'' directory, run ''make openmpi'', which should produce a ''lmp_openmpi'' executable.<br />
<br />
--[[User:sellan|sellan]] 14:08 Jul 28 2011<br />
<br />
'''Scaling Tests on GPC'''<br />
<br />
Results from strong scaling tests for LAMMPS using EAM potentials on GPC are shown in the graph on the right. Test simulation ran 500 timesteps for 4,000,000 atoms.<br />
<br />
Results from weak scaling tests for LAMMPS using EAM potentials on GPC are shown in the graph on the right. Test simulation ran 500 timesteps for 32,000 atoms per processor.<br />
<br />
OpenMPI version used: openmpi/1.4.1-intel-v11.0-ofed<br />
<br />
IntelMPI version used: intelmpi/impi-4.0.0.013<br />
<br />
LAMMPS version used: 15 Jan 2010<br />
<br />
'''Summary of Scaling Tests'''<br />
<br />
Results show good scaling for both OpenMPI and IntelMPI on Ethernet up to 16 processors, after which performance begins to suffer. On Infiniband, excellent scaling is maintained to 512 processors.<br />
<br />
IntelMPI shows slightly better performance compared to OpenMPI when running with Infiniband.<br />
<br />
--[[User:jchu|jchu]] 14:08 Feb 2, 2010<br />
<br />
===Monte Carlo (MC) simulation===</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=GPU_Benchmarks&diff=9396GPU Benchmarks2018-08-31T23:23:20Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
Instructions for running the Gromacs GPU Benchmark Single Node test case on Linux: <br />
<br />
1. Download [https://wiki.scinet.utoronto.ca/wiki/images/a/aa/GROMACS_GPU_BENCH.tar.gz Gromacs GPU Benchmark]<br />
<br />
2. Tar and unzip<br />
<br />
3. Run "build.sh" script with either "OPENCL" or "CUDA" <br />
<br />
4. Run benchmark script run.sh<br />
<br />
5. Report performance of ns/day on a single node.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Running_CCSM4&diff=9395Running CCSM42018-08-31T23:22:53Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
'''It is important to point out that all updates to the model system will only occur with CESM1.0 updates, not with CCSM4.0. It is also important to note that CCSM4 is a subset of CESM1. Although CESM1 supersedes CCSM4, users can run all CCSM4 experiments from the CESM1 code base.<br />
'''<br />
<br />
'''Note:''' To run interactively on GPC interactive queue or TCS on small subset of interactive processors, see the instructions below. <br />
<br />
<br />
The scientifically validated CESM1 runs are found in the list below (including a complete list of the model resolutions):<br />
<br />
/project/ccsm/cesm1_current/scripts/create_newcase --list<br />
<br />
<br />
--------------------------------------------------------------------------------<br />
CESM1.0 README<br />
<br />
For both a quick start as well as a detailed summary of creating and running <br />
a CESM model case, see the CESM1.0 User's Guide at<br />
http://www.cesm.ucar.edu/models/cesm1.0<br />
<br />
IMPORTANT INFORMATION ABOUT SCIENTIFIC VALIDATION<br />
<br />
CESM1.0 has the flexibility to configure cases with many different <br />
combinations of component models, grids, and model settings, but this <br />
version of CESM has only been validated scientifically for the following <br />
fully active configurations:<br />
<br />
1.9x2.5_gx1v6 B_1850_CN<br />
1.9x2.5_gx1v6 B_1850_RAMPCO2_CN<br />
1.9x2.5_gx1v6 B_1850-2000_CN<br />
<br />
1.9x2.5_gx1v6 B_1850_CAM5<br />
<br />
0.9x1.25_gx1v6 B_1850_CN<br />
0.9x1.25_gx1v6 B_1850_RAMPCO2_CN<br />
0.9x1.25_gx1v6 B_1850-2000_CN<br />
<br />
0.9x1.25_gx1v6 B_1850_BGC-BPRP<br />
0.9x1.25_gx1v6 B_1850_BGC-BDRD<br />
0.9x1.25_gx1v6 B_1850-2000_BGC-BPRP<br />
0.9x1.25_gx1v6 B_1850-2000_BGC-BDRD<br />
<br />
0.9x1.25_gx1v6 B_1850_CN_CHEM <br />
0.9x1.25_gx1v6 B_1850-2000_CN_CHEM<br />
<br />
1.9x2.5_gx1v6 B_1850_WACCM_CN<br />
1.9x2.5_gx1v6 B_1850-2000_WACCM_CN<br />
<br />
T31_gx3v7 B_1850_CN<br />
<br />
If the user is interested in running a "stand-alone" component configuration, <br />
the following model configurations have been validated scientifically and <br />
have associated diagnostic output as part of the release:<br />
<br />
1.9x2.5_1.9x2.5 F_2000_WACCM<br />
1.9x2.5_1.9x2.5 F_AMIP_CAM5<br />
1.9x2.5_1.9x2.5 F_AMIP_CN<br />
0.9x1.25_0.9x1.25 F_AMIP_CN<br />
<br />
0.9x1.25_gx1v6 I_2000<br />
0.9x1.25_gx1v6 I_2000_CN<br />
<br />
T62_gx1v6 C_NORMAL_YEAR<br />
<br />
For more information regarding alternative component configurations, <br />
please refer to the individual component web pages at<br />
http://www.cesm.ucar.edu/models/cesm1.0<br />
<br />
----<br />
CESM1 RESOLUTIONS: name (shortname) <br />
pt1_pt1 (pt1) <br />
0.23x0.31_0.23x0.31 (f02_f02) <br />
0.23x0.31_gx1v6 (f02_g16) <br />
0.23x0.31_tx0.1v2 (f02_t12) <br />
0.47x0.63_0.47x0.63 (f05_f05) <br />
0.47x0.63_gx1v6 (f05_g16) <br />
0.47x0.63_tx0.1v2 (f05_t12) <br />
0.9x1.25_0.9x1.25 (f09_f09) <br />
0.9x1.25_gx1v6 (f09_g16) <br />
1.9x2.5_1.9x2.5 (f19_f19) <br />
1.9x2.5_gx1v6 (f19_g16) <br />
4x5_4x5 (f45_f45) <br />
4x5_gx3v7 (f45_g37) <br />
T62_gx3v7 (T62_g37) <br />
T62_tx0.1v2 (T62_t12) <br />
T62_gx1v6 (T62_g16) <br />
T31_T31 (T31_T31) <br />
T31_gx3v7 (T31_g37) <br />
T42_T42 (T42_T42) <br />
10x15_10x15 (f10_f10) <br />
ne30np4_1.9x2.5_gx1v6 (ne30_f19_g16) <br />
ne240np4_0.23x0.31_gx1v6 (ne240_f02_g16) <br />
T85_T85 (T85_T85) <br />
<br />
COMPSETS: name (shortname): description <br />
A_PRESENT_DAY (A) <br />
Description: All data model <br />
A_GLC (AG) <br />
Description: All data model plus glc (glacier model) <br />
B_2000 (B) <br />
Description: All active components, present day <br />
B_2000_CN (BCN) <br />
Description: all active components, present day, with CN (Carbon Nitrogen) in clm <br />
B_1850_CAM5 (B1850C5) <br />
Description: All active components, pre-industrial, cam5 physics <br />
B_1850 (B1850) <br />
Description: All active components, pre-industrial <br />
B_1850_CN (B1850CN) <br />
Description: all active components, pre-industrial, with CN (Carbon Nitrogen) in CLM <br />
B_2000_CN_CHEM (B2000CNCHM) <br />
Description: All active components, pre-industrial, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm <br />
B_1850_CN_CHEM (B1850CNCHM) <br />
Description: All active components, pre-industrial, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm <br />
B_1850_RAMPCO2_CN (B1850RMCN) <br />
Description: All active components, pre-industirial with co2 ramp, with CN (Carbon Nitrogen) in CLM <br />
B_1850-2000 (B20TR) <br />
Description: All active components, 1850 to 2000 transient <br />
B_1850-2000_CN (B20TRCN) <br />
Description: All active components, 1850 to 2000 transient, with CN (Carbon Nitrogen) in CLM <br />
B_1850-2000_CN_CHEM (B20TRCNCHM) <br />
Description: All active components, 1850 to 2000 transient, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm <br />
B_1850-2000_CAM5 (B20TRC5) <br />
Description: All active components, 1850 to 2000 transient, cam5 physics <br />
B_2000_GLC (BG) <br />
Description: all active components, with active glc <br />
B_2000_TROP_MOZART (BMOZ) <br />
Description: All active components, with trop_mozart <br />
B_1850_WACCM (B1850W) <br />
Description: all active components, pre-industrial, with waccm <br />
B_1850_WACCM_CN (B1850WCN) <br />
Description: all active components, pre-industrial, with waccm and CN <br />
B_1850-2000_WACCM_CN (B20TRWCN) <br />
Description: All active components, 1850 to 2000 transient, WACCM with CN (Carbon Nitrogen) in CLM <br />
B_1955-2005_WACCM_CN (B55TRWCN) <br />
Description: All active components, 1955 to 2000 transient, WACCM with daily solar data and SPEs, CLM with CN <br />
B_1850_BGC-BPRP (B1850BPRP) <br />
Description: All active components, pre-industrial, CN in CLM, ECO in POP, BGC CO2=prog, rad CO2=prog <br />
B_1850_BGC-BDRD (B1850BDRD) <br />
Description: All active components, pre-industrial, CN in CLM, ECO in POP, BGC CO2=diag, rad CO2=diag <br />
B_1850-2000_BGC-BPRP (B20TRBPRP) <br />
Description: All active components, 1850 to 2000 transient, CN in CLM, ECO in POP, BGC CO2=prog, rad CO2=prog <br />
B_1850-2000_BGC-BDRD (B20TRBDRD) <br />
Description: All active components, 1850 to 2000 transient, CN in CLM, ECO in POP, BGC CO2=diag, rad CO2=diag <br />
C_NORMAL_YEAR_ECOSYS (CECO) <br />
Description: Active ocean model with ecosys and with COREv2 normal year forcing <br />
C_NORMAL_YEAR (C) <br />
Description: Active ocean model with COREv2 normal year forcing <br />
D_NORMAL_YEAR (D) <br />
Description: Active ice model with COREv2 normal year forcing <br />
E_2000 (E) <br />
Description: Fully active cam and ice with som ocean, present day <br />
E_2000_GLC (EG) <br />
Description: Fully active cam and ice with som ocean and glc, present day <br />
E_1850_CN (E1850CN) <br />
Description: Pre-industrial fully active ice and som ocean, with CN <br />
E_1850_CAM5 (E1850C5) <br />
Description: Pre-industrial fully active ice and som ocean, cam5 physics <br />
F_AMIP_CN (FAMIPCN) <br />
Description: AMIP run for CMIP5 protocol - valid only for 1 degree cam/clm/pres-cice <br />
F_AMIP_CAM5 (FAMIPC5) <br />
Description: AMIP run for CMIP5 protocol with cam5 <br />
F_1850 (F1850) <br />
Description: Pre-industrial cam/clm with prescribed ice/ocn <br />
F_1850_CAM5 (F1850C5) <br />
Description: Pre-industrial cam/clm with prescribed ice/ocn, cam5 physics <br />
F_2000 (F) <br />
Description: Stand-alone cam default, prescribed ocn/ice <br />
F_2000_CAM5 (FC5) <br />
Description: Stand-alone cam default, prescribed ocn/ice, cam5 physics <br />
F_2000_CN (FCN) <br />
Description: Stand-alone cam default, prescribed ocn/ice with CN <br />
F_1850-2000_CN (F20TRCN) <br />
Description: 20th Century transient stand-alone cam default, prescribed ocn/ice, with CN <br />
F_2000_GLC (FG) <br />
Description: Stand-alone cam default, prescribed ocn/ice, glc (glacier model) <br />
F_1850_CN_CHEM (F1850CNCHM) <br />
Description: stand-alone cam/clm, pre-industrial, with CN in CLM, super_fast_llnl chem in cam <br />
F_1850_WACCM (F1850W) <br />
Description: Pre-industrial cam/clm with prescribed ice/ocn <br />
F_2000_WACCM (FW) <br />
Description: present-day cam/clm with prescribed ice/ocn <br />
G_1850_ECOSYS (G1850ECO) <br />
Description: 1850 control for pop-ecosystem/cice/datm7/dlnd-rx1 <br />
G_NORMAL_YEAR (G) <br />
Description: Coupled ocean ice with COREv2 normal year forcing <br />
H_PRESENT_DAY (H) <br />
Description: Coupled ocean ice slnd <br />
I_2000 (I) <br />
Description: Active land model with QIAN atm input data for 2003 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000 <br />
I_1850 (I1850) <br />
Description: Active land model with QIAN atm input data for 1948 to 1972 and Satellite phenology (SP), CO2 level and Aerosol deposition for 1850 <br />
I_2000_GLC (IG) <br />
Description: Active glacier model and active land model with QIAN atm input data for 2003 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000 <br />
I_1948-2004 (I4804) <br />
Description: Active land model with QIAN atm input data for 1948 to 2004 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000 <br />
I_1850-2000 (I8520) <br />
Description: Active land model with QIAN atm input data for 1948 to 2004 and transient Satellite phenology (SP), and Aerosol deposition from 1850 to 2000 and 2000 CO2 level <br />
I_2000_CN (ICN) <br />
Description: Active land model with QIAN atm input data for 2003 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 2000 <br />
I_1850_CN (I1850CN) <br />
Description: Active land model with QIAN atm input data for 1948 to 1972 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 1850 <br />
I_1948-2004_CN (I4804CN) <br />
Description: Active land model with QIAN atm input data for 1948 to 2004 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 2000 <br />
I_1850-2000_CN (I8520CN) <br />
Description: Active land model with QIAN atm input data for 1948 to 1972 and transient CN, Aerosol dep from 1850 to 2000 and 2000 CO2 level <br />
S_PRESENT_DAY (S) <br />
Description: All stub models plus xatm <br />
X_PRESENT_DAY (X) <br />
Description: All dead model <br />
XG_PRESENT_DAY (XG) <br />
Description: All dead model and cism <br />
<br />
MACHINES: name (description)<br />
tcs (U of T IBM p6, os is AIX, 32 pes/node, batch system is Moab/LoadLeveler) <br />
gpc (U of T iDataPlex intel cluster, os is linux, 8 pes/node, batch system is Moab/Torque) <br />
bluefire (NCAR IBM p6, os is AIX, 32 pes/node, batch system is LSF) <br />
brutus_po (Brutus Linux Cluster ETH (pgi/9.0-1 with open_mpi/1.4.1), 16 pes/node, batch system LSF, added by UB) <br />
brutus_pm (Brutus Linux Cluster ETH (pgi/9.0-1 with mvapich2/1.4rc2), 16 pes/node, batch system LSF, added by UB) <br />
brutus_io (Brutus Linux Cluster ETH (intel/10.1.018 with open_mpi/1.4.1), 16 pes/node, batch system LSF, added by UB) <br />
brutus_im (Brutus Linux Cluster ETH (intel/10.1.018 with mvapich2/1.4rc2), 16 pes/node, batch system LSF, added by UB) <br />
edinburgh_lahey (NCAR CGD Linux Cluster (lahey), 8 pes/node, batch system is PBS) <br />
edinburgh_pgi (NCAR CGD Linux Cluster (pgi), 8 pes/node, batch system is PBS) <br />
edinburgh_intel (NCAR CGD Linux Cluster (intel), 8 pes/node, batch system is PBS) <br />
franklin (NERSC XT4, os is CNL, 4 pes/node, batch system is PBS) <br />
hadley (UCB Linux Cluster, os is Linux (ia64), batch system is PBS) <br />
hopper (NERSC XT5, os is CNL, 8 pes/node, batch system is PBS) <br />
intrepid (ANL IBM BG/P, os is BGP, 4 pes/node, batch system is cobalt) <br />
jaguar (ORNL XT4, os is CNL, 4 pes/node, batch system is PBS) <br />
jaguarpf (ORNL XT5, os is CNL, 12 pes/node, batch system is PBS) <br />
kraken (NICS/UT/teragrid XT5, os is CNL, 12 pes/node) <br />
lynx_pgi (NCAR XT5, os is CNL, 12 pes/node, batch system is PBS) <br />
midnight (ARSC Sun Cluster, os is Linux (pgi), batch system is PBS) <br />
pleiades (NASA/AMES Linux Cluster, Linux (ia64), Altix ICE, 3.0 GHz Harpertown processors, 8 pes/node and 8 GB of memory, batch system is PBS) <br />
pleiades_wes (NASA/AMES Linux Cluster, Linux (ia64), Altix ICE, 2.93 GHz Westmere processors, 12 pes/node and 24 GB of memory, batch system is PBS) <br />
prototype_atlas (LLNL Linux Cluster, Linux (pgi), 8 pes/node, batch system is Moab) <br />
prototype_hera (LLNL Linux Cluster, Linux (pgi), 16 pes/node, batch system is Moab) <br />
prototype_columbia (NASA Ames Linux Cluster, Linux (ia64), 2 pes/node, batch system is PBS) <br />
prototype_frost (NCAR IBM BG/L, os is BGL, 8 pes/node, batch system is cobalt) <br />
prototype_nyblue (SUNY IBM BG/L, os is BGL, 8 pes/node, batch system is cobalt) <br />
prototype_ranger (TACC Linux Cluster, Linux (pgi), 1 pes/node, batch system is SGE) <br />
prototype_ubgl (LLNL IBM BG/L, os is BGL, 2 pes/node, batch system is Moab) <br />
generic_ibm (generic ibm power system, os is AIX, batch system is LoadLeveler, user-defined) <br />
generic_xt (generic CRAY XT, os is CNL, batch system is PBS, user-defined) <br />
generic_linux_pgi (generic linux (pgi), os is Linux, batch system is PBS, user-defined) <br />
generic_linux_lahey (generic linux (lahey), os is Linux, batch system is PBS, user-defined) <br />
generic_linux_intel (generic linux (intel), os is Linux, batch system is PBS, user-defined) <br />
generic_linux_pathscale (generic linux (pathscale), os is Linux, batch system is PBS, user-defined) <br />
<br />
----<br />
<br />
<br />
''Initializing the Model Setup:''<br />
<br />
The initial setup of the model on TCS is simplified with the short script below<br />
<br />
#!/bin/bash<br />
<br />
export CCSMROOT=/project/ccsm/ccsm4_0_current<br />
export SCRATCH=/scratch/$USER<br />
export MACH=tcs<br />
export COMPSET=B_1850_CN<br />
export RES=f19_g16<br />
export CASEROOT=~/runs/ccsm4_comp-${COMPSET}_res-${RES}<br />
<br />
cd $CCSMROOT/scripts<br />
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES -scratchroot $SCRATCH <br />
<br />
'''NOTE:''' $CCSMROOT should point to the model code version in ''/project/ccsm'' with the "_current" after it. The same for CESM1<br />
<br />
'''NOTE:''' $CASEROOT which will specify your model run naming conventions will not archive properly (to short term archiving directory i.e. /scratch) if your CASEROOT name is too long. It is best to keep it short. <br />
<br />
<br />
<br />
<br />
This script creates an 1850 control with all components of the model fully active and carbon nitrogen cycling in the land component, The resolution is 1.9x2.5 in the atmosphere and x1 in the ocean. The file is created in the ~/run directory:<br />
<br />
For valid component sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/a2967.html <br />
For information on resolution sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/x42.html#ccsm_grids <br />
<br />
<br />
''Load Balancing:''<br />
<br />
For the NCAR bluefire load balancing table for a select set of simulations see:<br />
CESM1: http://www.cesm.ucar.edu/models/cesm1.0/timing/<br />
CCSM4: http://www.cesm.ucar.edu/models/ccsm4.0/timing/<br />
<br />
cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16<br />
<br />
edit env_mach_pes.xml<br />
<br />
<entry id="NTASKS_ATM" value="448" /> <br />
<entry id="NTHRDS_ATM" value="1" /> <br />
<entry id="ROOTPE_ATM" value="0" /> <br />
<br />
<entry id="NTASKS_LND" value="320" /> <br />
<entry id="NTHRDS_LND" value="1" /> <br />
<entry id="ROOTPE_LND" value="160" /> <br />
<br />
<entry id="NTASKS_ICE" value="64" /> <br />
<entry id="NTHRDS_ICE" value="1" /> <br />
<entry id="ROOTPE_ICE" value="0" /> <br />
<br />
<entry id="NTASKS_OCN" value="256" /> <br />
<entry id="NTHRDS_OCN" value="1" /> <br />
<entry id="ROOTPE_OCN" value="224" /> <br />
<br />
<entry id="NTASKS_CPL" value="224" /> <br />
<entry id="NTHRDS_CPL" value="1" /> <br />
<entry id="ROOTPE_CPL" value="0" /> <br />
<br />
<entry id="NTASKS_GLC" value="1" /> <br />
<entry id="NTHRDS_GLC" value="1" /> <br />
<entry id="ROOTPE_GLC" value="0" /> <br />
<br />
<entry id="PSTRID_ATM" value="1" /> <br />
<entry id="PSTRID_LND" value="1" /> <br />
<entry id="PSTRID_ICE" value="1" /> <br />
<entry id="PSTRID_OCN" value="1" /> <br />
<entry id="PSTRID_CPL" value="1" /> <br />
<entry id="PSTRID_GLC" value="1" /> <br />
<br />
<br />
Once this file is modified you can configure the case<br />
<br />
./configure -case<br />
<br />
You will notice that configure will change the file the you just edited and you can see the total processors used by the simulation (704 or 11 nodes in this case):<br />
<br />
<entry id="TOTALPES" value="704" /> <br />
<entry id="PES_LEVEL" value="1r" /> <br />
<entry id="MAX_TASKS_PER_NODE" value="64" /> <br />
<entry id="PES_PER_NODE" value="64" /> <br />
<entry id="CCSM_PCOST" value="-3" /> <br />
<entry id="CCSM_TCOST" value="0" /> <br />
<entry id="CCSM_ESTCOST" value="-3" /> <br />
<br />
'''Note:''' Rather than modifying the load balancing manually, NCAR has written a script that resides in your $CASE running directory that allows you to modify the individual component CPU allocation without playing with the env_mach_pes.xml file:<br />
<br />
To try a different configuration we might want 8 cpus running the OCN component continually and the remaining 24 cpus running atm on 24 then LND, ICE and CPL on 8 each. To set this up you enter;<br />
<br />
configure -cleanmach<br />
xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val 24<br />
xmlchange -file env_mach_pes.xml -id NTASKS_LND -val 8 <br />
xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val 8<br />
xmlchange -file env_mach_pes.xml -id ROOTPE_ICE -val 8<br />
xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val 8<br />
xmlchange -file env_mach_pes.xml -id ROOTPE_CPL -val 16<br />
xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val 8<br />
xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val 24<br />
configure -case<br />
<br />
Then build and resubmit<br />
<br />
The task geometry used by loadleveler on TCS is located in the file: ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run<br />
<br />
Ensure that the proper modules are loaded:<br />
<br />
Currently Loaded Modulefiles:<br />
1) ncl/5.1.1 3) netcdf/4.0.1_nc3 5) xlf/13.1<br />
2) nco/3.9.6 4) parallel-netcdf/1.1.1 6) vacpp/11.1<br />
<br />
Now compile the model with:<br />
<br />
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build<br />
<br />
One of the pre-processing steps in this build sequence is to fetch inputdat sets (initial and boundary conditions) from the NCAR SVN server. You may want to do this yourself before you build on the ''datamover1'' node if there is a large amount of initial condition data to transfer from the NCAR repository. ''datamover1'' has a high bandwidth connection to the outside.<br />
Note: We have most of the input data on /project/ccsm already so this step will not be required for the more common configurations.<br />
<br />
> ssh datamover1<br />
Last login: Wed Jul 7 16:38:14 2010 from tcs-f11n06-gpfs<br />
user@gpc-logindm01:~>cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16<br />
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16><br />
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>./check_input_data -inputdata /project/ccsm/inputdata -export<br />
Input Data List Files Found:<br />
./Buildconf/cam.input_data_list<br />
./Buildconf/clm.input_data_list<br />
./Buildconf/cice.input_data_list<br />
./Buildconf/pop2.input_data_list<br />
./Buildconf/cpl.input_data_list<br />
export https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc /project/ccsm/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc ..... success<br />
<br />
<br />
''Setting the Simulation Length:''<br />
<br />
The amount of time that you would like to run the model can be set by editing ''env_run.xml'' at anytime in the setup sequence<br />
<br />
<!--"if RESUBMIT is greater than 0, then case will automatically resubmit (integer) " --><br />
<entry id="RESUBMIT" value="10" /> <br />
<br />
<!--"sets the run length in conjuction with STOP_N and STOP_DATE, valid values: none,never,nsteps,nstep,nseconds,nsecond,nminutes,nminute,nhours,nhour,ndays,nday,nmonths,nmonth,nyears,nyear,date,ifdays0,end (char) " --><br />
<entry id="STOP_OPTION" value="nmonths" /> <br />
<br />
<!--"sets the run length in conjuction with STOP_OPTION and STOP_DATE (integer) " --><br />
<entry id="STOP_N" value="12" /> <br />
<br />
These settings will tell the model to checkpoint after each model year (12 months) and run for a total of 10 years (10 checkpoints)<br />
<br />
<br />
''Running CCSM4 on the Distributed System (TCS):''<br />
<br />
The model is now ready to be submitted to the TCS batch queue<br />
<br />
llsubmit ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run<br />
<br />
Once the model has run through a checkpoint timing information on the simulation will be found in:<br />
~/runs/ccsm4_comp-B_1850_CN_res-f19_g16/timing<br />
<br />
Standard output from the model can be followed during runtime by going to:<br />
/scratch/guido/ccsm4_comp-B_1850_CN_res-f19_g16/run<br />
and running<br />
tail -f <component_log_file><br />
<br />
The model will archive the NetCDF output in:<br />
/scratch/$USER/archive<br />
<br />
<br />
''Cloning Simulations''<br />
<br />
A useful command that allow for the setup of multiple runs quickly is the clone command. It allows for the cloning of a case quickly (so there is no need to run the setup script above every time)<br />
cd ~/runs<br />
/project/ccsm/ccsm4_0_current/scripts/create_clone -clone ccsm4_comp-B_1850_CN_res-f09_g16 -case ccsm4_comp-B_1850_CN_res-f09_g16_clone -v<br />
<br />
To change the load balancing (env_mach_pes.xml) in a current simulation setup or other parameters you can do a clean build to make sure the model is rebuilt properly:<br />
./configure -cleanmach<br />
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.clean_build<br />
./configure -case<br />
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build<br />
<br />
<br />
''Running CCSM4 on GPC''<br />
<br />
The setup script is almost identical:<br />
<br />
#!/bin/bash<br />
<br />
export CCSMROOT=/project/ccsm/ccsm4_0_current<br />
export SCRATCH=/scratch/guido<br />
export MACH=gpc<br />
export COMPSET=B_1850_CN<br />
export RES=f09_g16<br />
export CASEROOT=~/runs/ccsm4gpc_comp-${COMPSET}_res-${RES}<br />
<br />
cd $CCSMROOT/scripts<br />
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES <br />
<br />
To load balance and run the model follow the steps above:<br />
The env_mach_pes.xml configuration files needs to be modified as follows:<br />
<entry id="MAX_TASKS_PER_NODE" value="8" /> <br />
<entry id="PES_PER_NODE" value="8" /> <br />
<br />
Use qsub to submit the model to the GPC cluster:<br />
qsub ccsm4gpc_comp-B_1850_CN_res-f19_g16.tcs.run<br />
<br />
<br />
<br />
''Running CCSM4 interactively on TCS''<br />
<br />
In the run script you need to specify the following set and unset of environment variables. You also need a hostfile that will describe the node procs that you will be using<br />
<br />
unsetenv MP_EUILIB<br />
<br />
setenv MP_PROCS 16 <br />
<br />
setenv MP_NODES 1<br />
<br />
<br />
/usr/bin/poe /project/ccsm/bin/ccsm_launch ./ccsm.exe -hfile /project/<user>/runs/<casename>/hostfile<br />
<br />
and the hostfile looks like (for 16 threads of tcs02)<br />
<br />
--> more hostfile<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Installing_your_own_modules&diff=9394Installing your own modules2018-08-31T23:22:33Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
''Note: This page deals with modules for the 'modulecmd' package. For python modules go to [[Installing your own Python modules]]. For R modules go to [[R Statistical Package]]. For local perl modules, see [[Perl]].''<br />
<br />
=== Rationale ===<br />
There are many potential software packages that users could potentially want. These packages have potential conflicts and dependencies. For that reason, we install packages in nonstandard locations, and use the module command to load those locations into the appropriate environment variables. The usage of the module command is explained on the [[Software and Libraries]] page. <br />
<br />
It is possible for users to make their own collection of modules to supplement or replace the shared modules. Local user-space package and moduless are almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users package choices don't conflict. <br />
<br />
With local modules, you could load modules using <tt>module load [name]</tt>, just as system-wide modules. You'd have less clutter in your .bashrc, as paths can be set up in the module file, you can install and try out conflicting versions of software pages. <br />
<br />
Another nice feature of modules is that loading them twice does not append to the paths twice. So you could re-source your .bashrc file, and not clutter up the environment variables.<br />
<br />
=== How to do it ===<br />
<br />
* Type<br />
<pre><br />
$ module load use.own<br />
</pre><br />
This creates a directory <tt>$HOME/privatemodules</tt> if it doesnt exist yet, and adds it to the module path. It also make the man page for <tt>modulefile</tt> available.<br />
* Write module files, and put them in that directory (see below)<br />
* Put <tt>module load use.own</tt> in your <tt>.bashrc</tt><br />
* Use <tt>module load/unload/switch/avail/whatis/list</tt> on your local modules.<br />
<br />
=== Writing modulefiles ===<br />
<br />
Modulefiles use commands which are an extension of the standarde scripting language Tool Command Language (tcl). The name of the modulefile becomes the name of the module used in shell commands like <tt>module load [name]</tt>. You can create a hierarchy of modules (for instance for different versions) by using subdirectories in <tt>~/privatemodules</tt>. The module names then are the prepended by the relative path within <tt>~/privatemodules</tt>.<br />
<br />
In the modulefile, you can put commands to define or append environment variables and paths, express prerequirements and express conflicts.<br />
<br />
Note that you do not have to learn much tcl to use them though, as a simple example shows. The file <tt>~/privatemodules/somemodule</tt> could contain<br />
<source lang="bash"><br />
#%Module -*- tcl -*-<br />
## This is a module to access something<br />
proc ModulesHelp { } {<br />
puts stderr "This module sets up access to something" <br />
}<br />
module-whatis "sets up access to something"<br />
prereq somethingelse<br />
conflict thatothermodule<br />
module load gcc<br />
setenv SOMEVERION 0.95<br />
append-path PATH /home/[user]/[somedir]/bin<br />
append-path MANPATH /home/[user]/[somedir]/man<br />
append-path LD_LIBRARY_PATH /home/[user]/[somedir]/lib<br />
</source><br />
Explanation by line:<br />
* line 1: identifies this as a module file<br />
* line 2: is a comment that explains what the module does.<br />
* lines 3-5: is a function definition so that <tt>module help somemodule</tt> works<br />
* line 6 makes <tt>module whatis somemodule</tt> work<br />
* line 7 expresses a prerequist that the module <tt>somethingelse</tt> should be loaded before this one.<br />
* line 8 expresses a conflict, i.e., if <tt>thatothermodule</tt> is loaded, this one cannot be loaded as well.<br />
* lines 9 through 13 are the heart of the matter. They load another module, in this case, gcc, set an environment variable, and add directories to the paths.<br />
<br />
To learn more, simply type (if you have the use.own modules loaded):<br />
<pre><br />
$ man modulefile<br />
</pre><br />
<br />
=== Example ===<br />
<br />
An example of creating your own module can be found on the [[Brian]] page.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=User_Serial_on_GPC&diff=9393User Serial on GPC2018-08-31T23:22:00Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==Running serial jobs on SciNet's GPC==<br />
<br />
See [[User Serial]].</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=User_Serial&diff=9392User Serial2018-08-31T23:20:55Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
===General considerations===<br />
<br />
====Use whole nodes...====<br />
<br />
When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.<br />
<br />
Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.<br />
<br />
====...but not more.====<br />
<br />
When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available <br />
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])<br />
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconveniencing you and other researchers waiting for that node.<br />
<br />
If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.<br />
<br />
If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.<br />
<br />
====Is your job really serial?====<br />
<br />
While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below. <br />
<br />
If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.<br />
<br />
===Serial jobs of similar duration===<br />
<br />
The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a <br />
bit like this<br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque submission script for multiple serial jobs on<br />
# SciNet GPC<br />
#<br />
#PBS -l nodes=1:ppn=8,walltime=1:00:00<br />
#PBS -N serialx8<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
# Turn off implicit threading in Python, R<br />
export OMP_NUM_THREADS=1<br />
<br />
# EXECUTION COMMAND; ampersand off 8 jobs and wait<br />
(cd jobdir1; ./dojob1) &<br />
(cd jobdir2; ./dojob2) &<br />
(cd jobdir3; ./dojob3) &<br />
(cd jobdir4; ./dojob4) &<br />
(cd jobdir5; ./dojob5) &<br />
(cd jobdir6; ./dojob6) &<br />
(cd jobdir7; ./dojob7) &<br />
(cd jobdir8; ./dojob8) &<br />
wait<br />
</source><br />
<br />
There are four important things to take note of here. First, the <tt>'''wait'''</tt><br />
command at the end is crucial; without it the job will terminate <br />
immediately, killing the 8 programs you just started.<br />
<br />
Second is that it is important to group the programs by how long they <br />
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1, <br />
then for one hour 7 of the 8 cores on the GPC node are wasted; they are <br />
sitting idle but are unavailable for other users, and the utilization of <br />
this node over the whole run is only 56%. This is the sort of thing <br />
we'll notice, and users who don't make efficient use of the machine will <br />
have their ability to use scinet resources reduced. If you have many serial jobs of varying length, <br />
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].<br />
<br />
Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.<br />
<br />
===GNU Parallel===<br />
<br />
GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in<br />
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.<br />
<br />
GNU parallel is accessible on the GPC in the module<br />
<tt>gnu-parallel</tt>:<br />
<source lang="bash"><br />
module load gnu-parallel/20140622<br />
</source><br />
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version. <br />
<br />
The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.<br />
<br />
It is easiest to demonstrate the usage of GNU parallel by<br />
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC<br />
#<br />
#PBS -l nodes=1:ppn=8,walltime=24:00:00<br />
#PBS -N gnu-parallel-example<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
# Turn off implicit threading in Python, R<br />
export OMP_NUM_THREADS=1<br />
<br />
module load gnu-parallel/20140622 <br />
<br />
# EXECUTION COMMAND<br />
parallel -j 8 <<EOF<br />
cd jobdir1; ./dojob1; echo "job 1 finished"<br />
cd jobdir2; ./dojob2; echo "job 2 finished"<br />
cd jobdir3; ./dojob3; echo "job 3 finished"<br />
cd jobdir4; ./dojob4; echo "job 4 finished"<br />
cd jobdir5; ./dojob5; echo "job 5 finished"<br />
cd jobdir6; ./dojob6; echo "job 6 finished"<br />
cd jobdir7; ./dojob7; echo "job 7 finished"<br />
cd jobdir8; ./dojob8; echo "job 8 finished"<br />
cd jobdir9; ./dojob9; echo "job 9 finished"<br />
cd jobdir10; ./dojob10; echo "job 10 finished"<br />
cd jobdir11; ./dojob11; echo "job 11 finished"<br />
cd jobdir12; ./dojob12; echo "job 12 finished"<br />
cd jobdir13; ./dojob13; echo "job 13 finished"<br />
cd jobdir14; ./dojob14; echo "job 14 finished"<br />
cd jobdir15; ./dojob15; echo "job 15 finished"<br />
cd jobdir16; ./dojob16; echo "job 16 finished"<br />
EOF<br />
</source><br />
<br />
The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.<br />
<br />
===Serial jobs of varying duration===<br />
<br />
If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization. <br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque submission script for multiple, dynamically-run <br />
# serial jobs on SciNet GPC<br />
#<br />
#PBS -l nodes=1:ppn=8,walltime=1:00:00<br />
#PBS -N serialdynamic<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
# Turn off implicit threading in Python, R<br />
export OMP_NUM_THREADS=1<br />
<br />
module load gnu-parallel/20140622 <br />
<br />
# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh<br />
echo job*.sh | tr ' ' '\n' | parallel -j 8<br />
</source><br />
Notes:<br />
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.<br />
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.<br />
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option. <br />
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.<br />
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].<br />
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.<br />
* While on the command line, the option "--bar" can be nice to see the progress, when running as a job, you would not see this status bar. Consider using the "<tt>--joblog FILENAME</tt>" option, which writes the finished jobs into the file FILENAME.<br />
* The latter also keeps track of failed jobs, so you can later try to redo those with the same command, but with the option "--resume" added.<br />
<br />
===Version for more than 8 cores at once (still serial)===<br />
<br />
If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque submission script for multiple, dynamically-run <br />
# serial jobs on SciNet GPC<br />
#<br />
#PBS -l nodes=25:ppn=8,walltime=1:00:00<br />
#PBS -N serialdynamicMulti<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
# Turn off implicit threading in Python, R<br />
export OMP_NUM_THREADS=1<br />
<br />
module load gnu-parallel/20140622<br />
<br />
# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE<br />
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}<br />
</source><br />
Explanation:<br />
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.<br />
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.<br />
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.<br />
* So parallel will run the 800 commands:<br/>./myrun 1<br/>./myrun 2<br/>...<br/>./myrun 800<br />
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).<br />
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.<br />
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).<br />
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.<br />
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>. SciNet's gnu-parallel modules automatically transfer <tt>OMP_NUM_THREADS</tt>, and typical environment variables set by most modules.<br />
<br />
Notes:<br />
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.<br />
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs. <br />
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.<br />
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.<br />
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.<br />
* <span style="color:red;">The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all!</span><br>This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.<br />
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.<br />
<br />
===More on GNU parallel=== <br />
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]<br />
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/<br />
* Its man page can be found here http://www.gnu.org/software/parallel/man.html<br />
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.<br />
<br />
===GNU Parallel Reference===<br />
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.<br />
<br />
===Older scripts===<br />
<br />
Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.<br />
<br />
--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Perl&diff=9391Perl2018-08-31T23:20:21Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
==Availability on SciNet systems==<br />
Perl is installed on both the GPC (version 5.10.1) and the TCS (version 5.8.8). While perl can be run without the need to load any software modules,some of perl's functionality (especially installing modules from CPAN) relies on the GNU c compiler, so if you use perl on the GPC you most likely want to load the gcc module first:<br />
<code><br />
module load gcc<br />
</code><br />
<br />
''Note: if, as we encourage, you use the intel compilers on the gpc as well, you should load the intel module after the gcc module to ensure that the environment variable are set properly.''<br />
<br />
==Perl modules==<br />
One of the nice features of perl is the large number of perl modules that are out there. The Comprehensive Perl Archive Network (CPAN) is a large repository for these modules.<br />
<br />
''Note: perl modules are internal to perl are should not be confused with the [[Software_and_Libraries|software module system]]''.<br />
<br />
There are so many perl modules that we cannot possible maintain all of these. A number of module come with perl out-of-the-box and are therefore available. To find out what those are, you can use the command<br />
<pre><br />
$ perl -MFile::Find=find -MFile::Spec::Functions -Tlw -e 'find { wanted => sub { print canonpath $_ if /\.pm\z/ },no_chdir => 1 }, @INC'<br />
</pre><br />
<br />
==Local perl modules from CPAN==<br />
<br />
Users are generally responsible for installing any other modules they need (although we can always help). This poses a slight hurdle, as CPAN's package system would try to install these in system directories to which users have no access. Fortunately, there is an easy way to setup CPAN to use a local folder for perl libraries. <br />
<br />
In the following, the local directory for perl libraries will be assumed to be <tt>~/perl5</tt>. For the most recent version of CPAN (which is required to install several modules and to use the local modules), follow the steps outlined below.<br />
<br />
===Modifications to .bashrc===<br />
<br />
A number of paths need to be set for perl to find the local modules you will install.<br />
<pre><br />
export PERL_LOCAL_LIB_ROOT="$HOME/perl5"<br />
export PERL_MB_OPT="--install_base $HOME/perl5"<br />
export PERL_MM_OPT="INSTALL_BASE=$HOME/perl5"<br />
export PERL5LIB="$HOME/perl5/lib/perl5/x86_64-linux-thread-multi:$HOME/perl5/lib/perl5"<br />
export PATH="$HOME/perl5/bin:$PATH"<br />
</pre><br />
<br />
===Setting up a user-space CPAN modules directory===<br />
<br />
The following only works for the GPC (for now).<br />
<br />
1. First, since installing perl modules may involve compilation, be sure to be on a devel node (e.g. <tt>ssh gpc01</tt>). On the gpc, make sure that the gcc module is loaded (<tt>module load gcc</tt>).<br />
<br />
2. If this is not your first time using CPAN (for instance, you have tried before using it and found that you could not install any packages), you may have to remove your ~/.cpan directory. This will trigger a new configuration in the next step.<br />
<br />
3. Start a CPAN shell:<br />
<pre><br />
$ cpan<br />
</pre><br />
<br />
4. It will ask if you would like to get as much as possible configured automatically/ Say '''no'''. (In older versions of perl, 5.8.8, it would ask you whether you are ready for a manual configuration, to which you'd say yes or <enter>).<br />
<br />
5. It will now ask you a bunch of questions, almost all of which you can answer with the default (i.e. press <enter>), except for a few:<br />
<br />
* When the installation asks for ''Parameters for the 'perl Makefile.PL' command'', enter: <pre> LIB=~/perl5/lib/perl5 INSTALLMAN1DIR=~/perl5/man/man1 INSTALLMAN3DIR=~/perl5/man/man3<br />
</pre>(all on one line).<br />
* When it asks for ''Parameters for the 'perl Build.PL' command'', it is suggested '''not''' to add an '--install_base ' argument (put a space in instead).<br />
* When it asks ''If no urllist has been chosen yet, would you prefer CPAN.pm to connect to the built-in default sites without asking? '' say '''yes'''.<br />
* When it asks for the repository to use, you can pick what you want, but after choosing 'North America' and 'Canada', I chose site #6, for no particular reason.<br />
* Warning messages regarding YAML and AddHistory should be harmless.<br />
<br />
7. A module called <tt> local::lib </tt> is required to read the .bashrc environment variables properly. To install the latest <tt> local::lib </tt>, type <br />
<pre><br />
notest install local::lib <br />
</pre> <br />
in the CPAN shell and press <enter>. If you are not sure if you should do this, do it anyways, as it installs a bunch of modules that other modules you may be using likely depend on. <br />
<br />
8. As the module is installing, numerous prerequisites will also be installed (including an updated version of CPAN). Answer each with the default option.<br />
<br />
9. Type <tt> exit </tt> into the cpan shell.<br />
<br />
With version 5.8.8 (the version which we had with CentOS 5), this recipe would throw an error, and two more steps were required:<br />
:10. Restart the cpan shell. The shell will ask if you want to configure as much as possible automatically. Choose yes.<br />
:11. Install <tt> local::lib </tt> again as you did in step 7 and now everything should install without errors.<br />
<br />
===Another way to set up a user-space CPAN modules directory===<br />
<br />
''Note: the previous, more involved, instructions appear to be necessary to install local::lib and larger packages such as BioPerl. For some local packages, the instructions below may suffice.''<br />
<br />
The following works for me (ejspence) on GPC.<br />
<br />
<br />
1. First, since installing perl modules may involve compilation, be sure to be on a devel node (e.g. <tt>ssh gpc01</tt>). <br />
<br />
2. On the GPC, make sure that the gcc module is loaded.<br />
<pre><br />
module load gcc<br />
</pre><br />
<br />
3. I only set one environment variable.<br />
<pre><br />
export PERL5LIB=$HOME/lib/perl5:$HOME/lib<br />
</pre><br />
<br />
4. If this is not your first time using CPAN (for instance, you have tried before using it and found that you could not install any packages), you may have to remove your ~/.cpan directory. This will trigger a new configuration in the next step.<br />
<br />
5. Start a CPAN shell:<br />
<pre><br />
$ cpan<br />
</pre><br />
<br />
6. It will ask if you would like to get as much as possible configured automatically. Say '''yes'''. <br />
<br />
7. Now use cpan to set the default installation directory<br />
<pre><br />
cpan><br />
cpan> o conf makepl_arg INSTALL_BASE=$HOME<br />
cpan> o conf commit<br />
cpan><br />
</pre><br />
<br />
You should now be able to install packages using cpan.<br />
<br />
===Installing CPAN modules===<br />
<br />
Either from the CPAN shell or from the command line, you can now install packages quite simly by <tt>perl -MCPAN -e install <perlmodule> </tt> or simply <tt>cpan <perlmodule></tt>.<br />
<br />
===Using the modules===<br />
<br />
You can use local perl modules just as you would use system-wide perl modules.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=MATLAB&diff=9390MATLAB2018-08-31T23:20:00Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
We often get questions about running MATLAB on GPC. With a few exceptions for compilers and debuggers, SciNet does not purchase licenses for commercial software. As such, SciNet does not have a license for MATLAB, nor will it in the future. If users wish to run MATLAB they must supply their own license, or explore alternative options. This page gives information about the options for getting your MATLAB code to run, in recommended order. <br />
<br />
<br />
== Not using MATLAB ==<br />
<br />
Users can attempt to run MATLAB code using the open-source program Octave, accessible through the octave module. Though there are some differences between the two programs, Octave has been designed to interpret MATLAB code and can often be used in place of MATLAB. If your MATLAB code does not use some of the more-fancy MATLAB toolboxes, you may be able to get away with using Octave instead. Be sure to test your implementation in Octave thoroughly before committing to this option.<br />
<br />
It is worth observing that, while convenient for prototyping and running on a single workstation, there are reasons to avoid using MATLAB for larger HPC/ARC projects. These include the prohibitive license cost for large-scale work, poor performance at scale, and portability issues. If you can switch to a license-free option, such as Python, it may be worth the effort.<br />
<br />
<br />
== Creating a MATLAB executable ==<br />
<br />
If MATLAB must be used, you may be able to compile your MATLAB code into a stand-alone executable, and run this on a GPC compute node. The version of MATLAB being used will require a compiler license, and the compilation must be done on a Linux machine. Once the compilation is done, the executable can be copied to SciNet, and run using the MATLAB Compiler Runtime (MCR), which can be accessed using the MCR module. The MCR used must be the same version of MATLAB as the compiler. If the version of MCR that you need is not listed among the MCR module versions, contact us and we will install the version which you require.<br />
<br />
Here is an example script which uses the MCR:<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque submission script for SciNet GPC <br />
#<br />
#PBS -l nodes=1:ppn=8,walltime=1:00:00<br />
#PBS -N test_matlab<br />
#PBS -q debug<br />
<br />
# load modules (gcc is needed for MCR/2015a)<br />
module load Xlibraries gcc/5.2.0 MCR/2015a<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory from which the job was submitted<br />
cd $PBS_O_WORKDIR<br />
<br />
# Directory for the MCR to use to write temporary files. Use whatever directory you wish.<br />
mkdir -p $SCRATCH/temp<br />
export MCR_CACHE_ROOT=$SCRATCH/temp<br />
<br />
# EXECUTION COMMAND (note that the MATLAB script may require that LD_LIBRARY_PATH be added<br />
# to the script arguments). Note that, if the calculations are serial, you must bundle 8 such<br />
# calculations together for production runs!<br />
./run_myscript.sh $MCR:$LD_LIBRARY_PATH<br />
</source><br />
<br />
== Tunnelling to a license server ==<br />
<br />
If you have access to a non-SciNet MATLAB license server, and have installed MATLAB in your $HOME directory, you can setup your submission script to access the external license server. The following lines should be added to the beginning of your submission script:<br />
<br />
<pre><br />
PORT=XXX # port number of the license server<br />
LICENSE_IP=AAA.BBB.CCC.DDD # IP address of the license server<br />
ssh gpc01 -L${PORT}:${LICENSE_IP}:${PORT} -N &<br />
</pre><br />
<br />
This last line will tunnel the port from the compute node back to the license server, through gpc01. The port number and IP address of the licence server must be supplied by the system administrator of the license server.<br />
<br />
<br />
== Using a different Consortium ==<br />
<br />
Both [https://www.sharcnet.ca/my/software/show/54 Sharcnet] and [https://www.westgrid.ca/support/software/matlab Westgrid] have purchased different types of MATLAB licenses. Users can contact those consortia if they wish to attempt to run on those systems.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9389BGQ OpenFOAM2018-08-31T23:03:10Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script by typing the modules you want load inside the batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Typical OpenFOAM Applications on BG/Q ==<br />
A list of examples will be shared here. These sample cases are derived from applications that are run on BG/Q but changed for confidentiality reasons. It can guide new users for their specific use cases. Most of the information here is OpenFOAM specific, not BG/Q specific.<br />
<br />
=== Wind Flow Around Buildings ===<br />
This is a tutorial case that can be found in $FOAM_TUTORIALS/incompressible/simpleFoam/windAroundBuildings<br />
<br />
=== Rotational Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== LES Models in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== Multiphase Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can convert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells which allows users to fully utilize all 16 compute cores on a node.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Python&diff=9388Python2018-08-31T23:01:36Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
[http://www.python.org/ Python] is programming language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python. <br />
<br />
There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.<br />
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].<br />
<br />
__FORCETOC__ <br />
<br />
== Python on the GPC ==<br />
<br />
We currently have several versions of python installed, compiled against fast intel math libraries. To load the python modules, type the following commands:<br />
<br />
{|<br />
! Version<br />
! Command<br />
|-<br />
|2.7.2<br />
|<tt>module load gcc intel python</tt><br />
|-<br />
|2.7.3<br />
|<tt>module load gcc intel/13.1.1 python/2.7.3</tt><br />
|-<br />
|2.7.5<br />
|<tt>module load gcc intel/13.1.1 python/2.7.5</tt><br />
|-<br />
|2.7.8<br />
|<tt>module load intel/15.0.2 python/2.7.8</tt><br />
|-<br />
|2.7.11<br />
|<tt>module load anaconda2/4.0.0</tt><br />
|-<br />
|2.7.13<br />
|<tt>module load anaconda2/4.3.1</tt><br />
|-<br />
|3.3.4<br />
|<tt>module load gcc intel/14.0.1 python/3.3.4</tt><br />
|-<br />
|3.5.1<br />
|<tt>module load anaconda3/4.0.0</tt><br />
|-<br />
|3.6.1<br />
|<tt>module load anaconda3/4.4.0</tt><br />
|}<br />
<br />
== Modules installed system-wide ==<br />
<br />
Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users &mdash; eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.<br />
<br />
Below is a list of the packages currently installed system-wide.<br />
<br />
{| border="1" cellpadding="10" cellspacing="0"<br />
!{{Hl2}}| Module <br />
!{{Hl2}}| python/2.7.2 <br />
!{{Hl2}}| python/2.7.3 <br />
!{{Hl2}}| python/2.7.5 <br />
!{{Hl2}}| python/2.7.8<br />
!{{Hl2}}| python/3.3.4<br />
!{{Hl2}}| Comments<br />
|- <br />
|[http://www.scipy.org/ SciPy]<br />
| 0.10.0<br />
| 0.11.0<br />
| 0.14.0<br />
| 0.14.0<br />
| 0.14.0<br />
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries. <br />
|-<br />
|[http://numpy.scipy.org/ NumPy]<br />
| 1.6.1<br />
| 1.7.0<br />
| 1.7.0<br />
| 1.9.1<br />
| 1.8.1<br />
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.<br />
|-<br />
| [http://mpi4py.scipy.org/ mpi4py]<br />
| 1.2.2<br />
| 1.2.2<br />
| 1.2.2<br />
| 1.2.2<br />
| 1.2.2<br />
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)<br />
|-<br />
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]<br />
| 2.0<br />
| 2.0.1<br />
| 2.2.1<br />
| 2.4<br />
| 2.4_rc2<br />
| Fast, memory-efficient elementwise operations on Numpy arrays.<br />
|-<br />
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]<br />
| 2.8 <br />
| -<br />
| -<br />
| -<br />
| -<br />
| A collection of scientific python utilities. Does not include MPI support. No longer supported.<br />
|-<br />
| [http://yt.enzotools.org/ yt]<br />
| 2.2<br />
| 2.5.3<br />
| 2.5.5<br />
| -<br />
| -<br />
| A collection of python tools for analyzing astrophysical simulation output.<br />
|-<br />
| [http://ipython.scipy.org/moin/ iPython]<br />
| 0.11 <br />
| 0.13.1<br />
| 1.0.0<br />
| 2.3.0<br />
| 1.2.1<br />
| An enhanced interactive python.<br />
|-<br />
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab<br />
| 1.1.0<br />
| 1.2.0<br />
| 1.3.0<br />
| 1.4.2<br />
| 1.3.1<br />
| Matlab-like plotting for python.<br />
|-<br />
| [http://www.pytables.org/moin PyTables]<br />
| 2.3.1 <br />
| 2.4.0<br />
| 3.0.0<br />
| 3.1.1<br />
| 3.1.1<br />
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. <br />
|-<br />
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]<br />
| 0.9.8<br />
| 1.0.4<br />
| 1.1.1<br />
| -<br />
| 1.1.0<br />
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. <br />
|-<br />
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]<br />
| 1.4.1<br />
| -<br />
| -<br />
| -<br />
| -<br />
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module. No longer supported.<br />
|-<br />
| [http://alfven.org/wp/hdf5-for-python/ h5py]<br />
| 2.0.1<br />
| 2.1.3<br />
| 2.2.0<br />
| 2.3.1<br />
| 2.3.0<br />
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.<br />
|-<br />
| [http://pysvn.tigris.org/ PySVN]<br />
| 1.7.1<br />
| -<br />
| -<br />
| -<br />
| -<br />
| Python interface to the svn version control system. <br />
|-<br />
| [http://mercurial.selenic.com/ Mercurial]<br />
| 2.0.1<br />
| 2.6.2<br />
| 2.7.1<br />
| 3.2<br />
| -<br />
| A distributed version-control system written in Python.<br />
|-<br />
| [http://cython.org/ Cython]<br />
| 0.15.1<br />
| 0.18<br />
| 0.19.1<br />
| 0.21.1<br />
| 0.20.1<br />
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.<br />
|-<br />
| [http://code.google.com/p/python-nose/ nose]<br />
| 1.1.2<br />
| 1.2.1<br />
| 1.3.0<br />
| 1.3.4<br />
| 1.3.0<br />
| A unit-testing framework for python.<br />
|- <br />
| [http://pypi.python.org/pypi/setuptools setuptools]<br />
| 0.6c11<br />
| 0.6c11<br />
| 1.1<br />
| 7.0<br />
| 5.1<br />
| Enables easy installation of new python modules<br />
|-<br />
| [http://pandas.pydata.org/ pandas]<br />
| 0.13.0<br />
| 0.13.0<br />
| 0.13.0<br />
| 0.15.0<br />
| 0.14.1<br />
| high-performance, easy-to-use data structures and data analysis tools.<br />
|-<br />
| [http://www.astropy.org astropy]<br />
| -<br />
| -<br />
| 0.3<br />
| 0.4.2<br />
| 0.3.2<br />
| astronomical routines<br />
|-<br />
| [http://briansimulator.org/ brian]<br />
| 1.4.1<br />
| 1.4.1<br />
| 1.4.1<br />
| 1.4.1<br />
| -<br />
| spiking neural network simulator<br />
|- <br />
|}<br />
<br />
<br />
== Producing Matplotlib Figures on GPC Compute Nodes and in Job Scripts ==<br />
<br />
The conventional way of producing figures from python using matplotlib i.e., <br />
<br />
import matplotlib.pyplot as plt<br />
plt.plot(.....)<br />
plt.savefig(...)<br />
<br />
will not work on the GPC compute nodes. The reason is that pyplot will try to open the figure in a window on the screen, but the compute nodes do not have screens or window managers. There is an easy workaround, however, that sets up a different 'backend' to matplotlib, one that does not try to open a window, as follows:<br />
<br />
import matplotlib as mpl<br />
mpl.use('Agg')<br />
import matplotlib.pyplot as plt<br />
plt.plot(.....)<br />
plt.savefig(...)<br />
<br />
It is essential that the <tt>mpl.use('Agg')</tt> command precedes the importing of pyplot. <br />
<br />
== Installing your own Python Modules ==<br />
<br />
Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict. <br />
<br />
To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.<br />
<br />
* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.<br />
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:<br />
<source lang=bash><br />
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/<br />
</source><br />
<br />
* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.<br />
<br />
* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,<br />
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing: <br />
<source lang=bash><br />
easy_install --prefix=${HOME} -O1 [packagename]<br />
</source> <br />
<br />
** Continue doing this until all of the packages you need to install are successfully installed.<br />
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:<br />
<source lang=bash><br />
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]<br />
</source> <br />
<br />
* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:<br />
** Download the relevant files<br />
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.<br />
** cd into the newly created directory, and run <br />
<source lang=bash><br />
python setup.py install --prefix=${HOME}<br />
</source> <br />
<br />
* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,<br />
<source lang=bash><br />
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg<br />
</source><br />
<br />
* You should now be done! Now, re-source your .bashrc and test your new python modules.<br />
<br />
* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules). <br />
<br />
[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=GPC_MPI_Versions&diff=9387GPC MPI Versions2018-08-09T17:32:22Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
You can only use ONE mpi version at a time as most versions use the same <br />
names for the mpirun and compiler wrappers, so be careful which modules you <br />
have loaded in your <tt>~/.bashrc</tt>. For this reason, we recommend generally not loading any modules in your .bashrc.<br />
<br />
__TOC__<br />
<br />
===OpenMPI===<br />
<br />
To use [http://www.open-mpi.org/ OpenMPI] compiled with the intel compilers load the modules <br />
<br />
<pre><br />
module load intel openmpi<br />
</pre> <br />
<br />
or for the gcc version use<br />
<br />
<pre><br />
module load gcc openmpi/1.4.2-gcc-v4.4.0-ofed <br />
</pre> <br />
<br />
The MPI library wrappers for compiling are mpicc/mpicxx/mpif90/mpif77.<br />
<br />
OpenMPI has been built to support various communication methods and automatically uses the best method<br />
depending on how and where it is run. To explicitly specify the method you can use the following <tt>--mca</tt><br />
flags following for infiniband<br />
<br />
<pre><br />
mpirun --mca pml ob1 --mca btl self,sm,openib -np 16 ./a.out<br />
</pre><br />
<br />
and the following for IPoIB (not recommended)<br />
<pre><br />
mpirun --mca pml ob1 --mca btl self,sm,tcp --mca btl_tcp_if_exclude lo,eth0 -np 16 ./a.out<br />
</pre><br />
<br />
If your program runs into what appears to be memory issues with larger jobs try using the xrc communications<br />
<pre><br />
mpirun --mca pml ob1 --mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 --mca btl_openib_max_send_size 12288" <br />
</pre><br />
<br />
If you are still having issues you can try openmpi > 1.6 which uses MellanoX Messaging library (MXM) by default over 128 mpi tasks<br />
<pre><br />
module load openmpi/intel/1.6.4<br />
</pre><br />
<br />
or can be explicitly asked for by using if using less than 128 mpi tasks<br />
<pre><br />
mpirun--mca mtl mxm --mca mtl_mxm_np 2 -np 16 ./a.out<br />
</pre><br />
<br />
To NOT use MXM, regardless the number of mpi tasks use:<br />
<br />
<pre><br />
mpirun --mca pml ob1 -np 16 ./a.out<br />
</pre><br />
<br />
<br />
For mixed openMP/MPI applications, set OMP_NUM_THREADS to the number of threads per process and add '<tt>--bynode</tt>' to the mpirun command, e.g.,<br />
<pre><br />
export OMP_NUM_THREADS=4<br />
mpirun -np 6 --bynode ./a.out<br />
</pre><br />
would start 6 MPI processes on different nodes, each with 4 openMP threads. If your script requests 3 nodes, each node gets 2 MPI processes. <br />
<br />
For more information on available flags see the OpenMPI [http://www.open-mpi.org/faq/ FAQ]<br />
<br />
===IntelMPI===<br />
<br />
[http://software.intel.com/en-us/intel-mpi-library/ IntelMPI] is also a [http://www.mcs.anl.gov/research/projects/mpich2/ MPICH2] derivative<br />
customized by Intel. To use IntelMPI (4.x) compiled with the intel compilers load the modules <br />
<br />
<pre><br />
module load intel intelmpi<br />
</pre> <br />
<br />
The MPI library wrappers for compiling are mpicc/mpicxx/mpif90/mpif77.<br />
<br />
IntelMPI requires a .mpd.conf file containing the variable "MPD_SECRETWORD=..." in your $HOME directory. To create this file use<br />
<br />
<pre><br />
echo "MPD_SECRETWORD=ABC123" > ~/.mpd.conf<br />
chmod 600 ~/.mpd.conf<br />
</pre><br />
<br />
IntelMPI, like OpenMPI, has been built to support various communication methods and automatically uses the best method<br />
depending on how and where it is run. <br />
<br />
<pre><br />
mpirun -np 16 ./a.out<br />
</pre><br />
<br />
<br />
To explicitly specify the method you can use the following flags to use <br />
the following flags for infiniband (dapl) and shared memory (best performance on IB)<br />
<br />
<pre><br />
mpirun -np 16 -env I_MPI_FABRICS=shm:dapl ./a.out<br />
</pre><br />
<br />
or the following for openfabrics (ibverbs) infiniband<br />
<pre><br />
mpirun -np 16 -genv I_MPI_FABRICS=shm:ofa ./a.out<br />
</pre><br />
<br />
or the following for IPoIB<br />
<br />
<pre><br />
mpirun -np 16 -env I_MPI_TCP_NETMASK=ib -env I_MPI_FABRICS shm:tcp ./a.out<br />
</pre><br />
<br />
If you run into communication errors that appear related to memory try some of the following flags<br />
<pre><br />
module load intelmpi/4.0.3.008<br />
mpirun -np 16 -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_OFA_NUM_RDMA_CONNECTIONS=-1 ./a.out<br />
</pre><br />
<br />
For hybrid openMP/MPI runs, set <tt>OMP_NUM_THREADS</tt> to the desired number of OpenMP threads per MPI process and specify the number of MPI processes per node on the mpirun command line with <tt>-ppn <num></tt>. E.g.<br />
<pre><br />
export OMP_NUM_THREADS=4<br />
mpirun -ppn 2 -np 6 ./a.out<br />
</pre><br />
would start a total of 6 MPI processes each with 4 threads, with each node running 2 MPI processes. Your script should request 3 nodes in this case.<br />
''Note: to compile for hybrid openMP/MPI with IntelMPI, you need to add the flag <tt>-mt_mpi</tt> to your compilation command (i.e. mpicc/mpif90/mpicxx).''<br />
<br />
For more information on these an other flags see Intel's [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/ Documentation]<br />
page especially the "Getting Started (Linux)" Guide.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Data_Management&diff=9386Data Management2018-08-09T17:32:13Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
=='''Storage Space'''==<br />
SciNet's storage system is based on IBM's [http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System GPFS] (General Parallel File System). There are two main systems for user data: <tt>/home</tt>, a small, backed-up space where user home directories are located, and <tt>/scratch</tt>, a large system for input or output data for jobs; data on <tt>/scratch</tt> is not only not backed up (a third storage system, /project, exist only for groups with LRAC/NRAC allocations). Data placed on scratch will be deleted if it has not been accessed in 3 months. SciNet does not provide long-term storage for large data sets. <br />
<br />
===Overview of the different file systems===<br />
<br />
{|border="1" cellpadding="10" cellspacing="0"<br />
|-<br />
! {{Hl2}} | file system <br />
! {{Hl2}} | purpose <br />
! {{Hl2}} | user quota <br />
! {{Hl2}} | block size <br />
! {{Hl2}} | backed up<br />
! {{Hl2}} | purged<br />
! {{Hl2}} | access <br />
|- <br />
| /home<br />
| development<br />
| 50 GB<br />
| 256 KB<br />
| yes<br />
| never<br />
| read-only on compute nodes (r/w on login, devel and datamover1) <br />
|- <br />
| /scratch<br />
| computation<br />
| first of (20 TB ; 1 million files)<br />
| 4 MB<br />
| no<br />
| files > 3 months<br />
| read/write on all nodes<br />
|- <br />
| /project<br />
| computation<br />
| by allocation<br />
| 256 KB<br />
| yes<br />
| never<br />
| read/write on all nodes <br />
|}<br />
project is included in scratch<br />
<br />
===Home Disk Space===<br />
<br />
Every SciNet user gets a 50GB directory on <tt>/home</tt> in a directory <tt>/home/G/GROUP/USER</tt>, which is regularly backed-up. Home is visible from <tt>login.scinet</tt> nodes, and from the development nodes on [[GPC_Quickstart | GPC]] and the [[TCS_Quickstart | TCS]]. However, on the compute nodes of the GPC clusters -- as when jobs are running -- <tt>/home</tt> is mounted '''''read-only'''''; thus GPC jobs can read files in /home but cannot write to files there. <tt>/home</tt> is a good place to put code, input files for runs, and anything else that needs to be kept to reproduce runs. On the other hand, <tt>/home</tt> is not a good place to put many small files, since<br />
the block size for the file system is 256KB, so you would quickly run out of disk quota and you will make the backup system very slow.<br />
<br />
If your application absolutely insists on writing material to your home account and you can't find a way to instruct it to write somewhere else, an alternative is to create a link pointing from your account under /home to a location under /scratch.<br />
<br />
===Scratch Disk Space===<br />
<br />
Every SciNet user also gets a directory in <tt>/scratch</tt> called <tt>/scratch/G/GROUP/USER</tt>. Scratch is visible from the <tt>login.scinet</tt> nodes, the development nodes on [[GPC_Quickstart | GPC]] and the [[TCS_Quickstart | TCS]], and on the compute nodes of the clusters, mounted as read-write. Thus jobs would normally write their output somewhere in <tt>/scratch</tt>. There are '''NO''' backups of anything on <tt>/scratch</tt>.<br />
<br />
There is a large amount of space available on <tt>/scratch</tt> but it is purged routinely so that all users running jobs and generating large outputs will have room to store their data temporarily. Computational results which you want to keep longer than this must be copied (using <tt>scp</tt>) off of SciNet entirely and to your local system. SciNet does not routinely provide long-term storage for large data sets. <br />
<br />
Also note that the shared parallel file system was not designed to do many small file transactions. For that reason, the number of files that any user can have on scratch is limited to 1 million. This limit should be thought of as a safeguard, not an invitation to create one million files. Please see [[File System and I/O dos and don'ts]]. <br />
<br />
===Scratch Disk Purging Policy===<br />
<br />
In order to ensure that there is always significant space available for running jobs '''we automatically delete files in /scratch that have not been accessed or modified for more than 3 months by the actual deletion day on the 15th of each month'''. Note that we recently changed the cut out reference to the ''MostRecentOf(atime,mtime)''. This policy is subject to revision depending on its effectiveness. More details about the purging process and how users can check if their files will be deleted follows. If you have files scheduled for deletion you should move them to a more permanent locations such as your departmental server or your /project space (for PIs who have either been allocated disk space by the LRAC or have bought diskspace).<br />
<br />
On the '''first''' of each month, a list of files scheduled for purging is produced, and an email notification is sent to each user on that list. Furthermore, at/or about the '''12th''' of each month a 2nd scan produces a more current assessment and another email notification is sent. This way users can double check that they have indeed taken care of all the files they needed to relocate before the purging deadline. Those files will be automatically deleted on the '''15th''' of the same month unless they have been accessed or relocated in the interim. If you have files scheduled for deletion then they will be listed in a file in /scratch/t/todelete/current, which has your userid and groupid in the filename. For example, if user xxyz wants to check if they have files scheduled for deletion they can issue the following command on a system which mounts /scratch (e.g. a scinet login node): '''ls -1 /scratch/t/todelete/current |grep xxyz'''. In the example below, the name of this file indicates that user xxyz is part of group abc, has 9,560 files scheduled for deletion and they take up 1.0TB of space:<br />
<br />
<pre><br />
[xxyz@scinet04 ~]$ ls -1 /scratch/t/todelete/current |grep xxyz<br />
-rw-r----- 1 xxyz root 1733059 Jan 12 11:46 10001___xxyz_______abc_________1.00T_____9560files<br />
</pre><br />
<br />
The file itself contains a list of all files scheduled for deletion (in the last column) and can be viewed with standard commands like more/less/cat - e.g. '''more /scratch/t/todelete/current/10001___xxyz_______abc_________1.00T_____9560files'''<br />
<br />
Similarly, you can also verify all other users on your group by using the ls command with grep on your group. For example: '''ls -1 /scratch/t/todelete/current |grep abc'''. That will list all other users in the same group that xxyz is part of, and have files to be purged on the 15th. Members of the same group have access to each other's contents.<br />
<br />
'''NOTE:''' Preparing these assessments takes several hours. If you change the access/modification time of a file in the interim, that will not be detected until the next cycle. A way for you to get immediate feedback is to use the ''''ls -lu'''' command on the file to verify the atime and ''''ls -la'''' for the mtime. If the file atime/mtime has been updated in the meantime, coming the purging date on the 15th it will no longer be deleted.<br />
<br />
===Project Disk Space===<br />
<br />
Investigators who have been granted allocations through the [http://wiki.scinethpc.ca/wiki/index.php/Application_Process LRAC/NRAC Application Process] may have been allocated disk space in addition to compute time. For the period of time that the allocation is granted, they will have disk space on the <tt>/project</tt> disk system. Space on project is a subset of scratch, but is not purged and is backed up. All members of the investigators groups will have access to this disk system, which will be mounted read/write everywhere.<br />
<br />
===How much Disk Space Do I have left?===<br />
<br />
The <tt>'''/scinet/gpc/bin6/diskUsage'''</tt> command, available on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.<br />
<pre><br />
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]<br />
-h|-?: help<br />
-a: list usages of all members on the group<br />
-u <user>: as another user on your group<br />
-de: include delta information<br />
-plot: create plots of disk usages<br />
</pre><br />
<br />
Did you know that you can check which of your directories have more than 1000 files with the <tt>'''/scinet/gpc/bin6/topUserDirOver1000list'''</tt> command and which have more than 1GB of material with the <tt>'''/scinet/gpc/bin6/topUserDirOver1GBlist'''</tt> command?<br />
<br />
Notes:<br />
* information on usage and quota is only updated hourly!<br />
* contents of project count against space and #files limits on scratch<br />
<br />
===Performance===<br />
<br />
[http://en.wikipedia.org/wiki/IBM_General_Parallel_File_System GPFS] is a high-performance filesystem which provides rapid reads and writes to large datasets in parallel from many nodes. As a consequence of this design, however, '''the file system performs quite ''poorly'' at accessing data sets which consist of many, small files.''' For instance, you will find that reading data in from one 16MB file is enormously faster than from 400 40KB files. Such small files are also quite wasteful of space, as the blocksize for the scratch filesystem is 16MB. This is something you should keep in mind when planning your input/output strategy for runs on SciNet.<br />
<br />
For instance, if you run multi-process jobs, having each process write to a file of its own is not an scalable I/O solution. A directory gets locked by the first process accessing it, so all other processes have to wait for it. Not only has the code just become considerably less parallel, chances are the file system will have a time-out while waiting for your other processes, leading your program to crash mysteriously.<br />
Consider using MPI-IO (part of the MPI-2 standard), which allows files to be opened simultaneously by different processes, or using a dedicated process for I/O to which all other processes send their data, and which subsequently writes this data to a single file.<br />
<br />
===Local Disk===<br />
<br />
The compute nodes on the GPC '''do not contain hard drives''' so there is no local disk available to use during your computation. You can however use part of a compute nodes RAM like a local disk ('ramdisk') but this will reduce how much memory is available for your<br />
program. This can be accessed using <tt>/dev/shm/</tt> and is currently set to 8GB. Anything written<br />
to this location that you want to keep must be copied back to the <tt>/scratch</tt> filesystem as <tt>/dev/shm</tt> is wiped after each job and since it is in memory will not survive through a reboot of the node. More on ramdisk usage can be found [[User_Ramdisk | here]].<br />
<br />
Note that the absense of hard drives also means that the nodes cannot swap memory, so be sure that your computation fits within memory.<br />
<br />
===Buying storage space on GPFS or HPSS===<br />
<br />
Groups can buy space on GPFS or HPSS rather than rely on the [http://wiki.scinethpc.ca/wiki/index.php/Application_Process annual allocation process]. A good budgetary number would be:<br />
<br />
GPFS $400/TB<br />
<br />
HPSS $150/TB<br />
<br />
This is a one-time cost. We have no formal, written data retention policy at this point but the intent is to keep any HPSS data (including migrating to new tape technologies) as long as SciNet is in operation. These numbers are for budgetary purposes only and subject to change (e.g. as markets and technologies evolve).<br />
<br />
<br />
=='''Data Transfer'''==<br />
{{:Data_Transfer}}<br />
<br />
=='''File/Ownership Management (ACL)'''==<br />
* By default, at SciNet, users within the same group have read permission to each other's files (not write)<br />
* You may use access control list ('''ACL''') to allow your supervisor (or another user within your group) to manage files for you (i.e., create, move, rename, delete), while still retaining your access and permission as the original owner of the files/directories.<br />
* '''NOTE''': We highly recommend that you never give write permission to other users on the top level of your home directory (/home/G/GROUP/[owner]), since that would seriously compromise your privacy, in addition to disable ssh key authentication, among other things. If necessary, make specific sub-directories under your home directory so that other users can manipulate/access files from those.<br />
* If you need to set up permissions across groups [mailto:support@scinet.utoronto.ca contact us] (and the other group's supervisor!).<br />
<br />
<!--<br />
<br />
===Using setfacl/getfacl===<br />
* To allow [supervisor] to manage files in /project/g/group/[owner] using '''setfacl''' and '''getfacl''' commands, follow the 3-steps below as the [owner] account from a shell:<br />
<pre><br />
1) $ /scinet/gpc/bin/setfacl -d -m user:[supervisor]:rwx /project/g/group/[owner]<br />
(every *new* file/directory inside [owner] will inherit [supervisor] ownership by default from now on)<br />
<br />
2) $ /scinet/gpc/bin/setfacl -d -m user:[owner]:rwx /project/g/group/[owner]<br />
(but will also inherit [owner] ownership, ie, ownership of both by default, for files/directories created by [supervisor])<br />
<br />
3) $ /scinet/gpc/bin/setfacl -Rm user:[supervisor]:rwx /project/g/group/[owner]<br />
(recursively modify all *existing* files/directories inside [owner] to also be rwx by [supervisor])<br />
<br />
$ /scinet/gpc/bin/getfacl /project/g/group/[owner]<br />
(to determine the current ACL attributes)<br />
<br />
$ /scinet/gpc/bin/setfacl -b /project/g/group/[owner]<br />
(to remove any previously set ACL)<br />
<br />
PS: on the datamovers getfacl, setfacl and chacl will be on your path<br />
</pre><br />
For more information on using [http://linux.die.net/man/1/setfacl <tt>setfacl</tt>] or [http://linux.die.net/man/1/getfacl <tt>getfacl</tt>] see their man pages.<br />
<br />
--><br />
<br />
===Using mmputacl/mmgetacl===<br />
* You may use gpfs' native '''mmputacl''' and '''mmgetacl''' commands. The advantages are that you can set "control" permission and that [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.doc%2Fgpfs31%2Fbl1adm1160.html POSIX or NFS v4 style ACL] are supported. You will need first to create a /tmp/supervisor.acl file with the following contents:<br />
<pre><br />
user::rwxc<br />
group::----<br />
other::----<br />
mask::rwxc<br />
user:[owner]:rwxc<br />
user:[supervisor]:rwxc<br />
</pre><br />
<br />
Then issue the following 2 commands:<br />
<pre><br />
1) $ mmputacl -i /tmp/supervisor.acl /project/g/group/[owner]<br />
2) $ mmputacl -d -i /tmp/supervisor.acl /project/g/group/[owner]<br />
(every *new* file/directory inside [owner] will inherit [supervisor] ownership by default as well as <br />
[owner] ownership, ie, ownership of both by default, for files/directories created by [supervisor])<br />
<br />
$ mmgetacl /project/g/group/[owner]<br />
(to determine the current ACL attributes)<br />
<br />
$ mmdelacl -d /project/g/group/[owner]<br />
(to remove any previously set ACL)<br />
<br />
$ mmeditacl /project/g/group/[owner]<br />
(to create or change a GPFS access control list)<br />
(for this command to work set the EDITOR environment variable: export EDITOR=/usr/bin/vi)<br />
</pre><br />
<br />
NOTES:<br />
* There is no option to recursively add or remove ACL attributes using a gpfs built-in command to existing files. You'll need to use the -i option as above for each file or directory individually. [[Data_Management#bash_script_that_you_may_adapt_to_recursively_add_or_remove_ACL_attributes_using_gpfs_built-in_commands | Here is a sample bash script you may use for that purpose]]<br />
<br />
* mmputacl/setfacl will not overwrite the original linux group permissions for a directory when copied to another directory already with ACLs, hence the "#effective:r-x" note you may see from time to time with mmgetacf/getfacl. If you want to give rwx permissions to everyone in your group you should simply rely on the plain unix 'chmod g+rwx' command. You may do that before or after copying the original material to another folder with the ACLs.<br />
<br />
For more information on using [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.doc%2Fgpfs31%2Fbl1adm11120.html <tt>mmputacl</tt>] or [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.doc%2Fgpfs31%2Fbl1adm11120.html <tt>mmgetaclacl</tt>] see their man pages.<br />
<br />
===Appendix (ACL)===<br />
====bash script that you may adapt to recursively add or remove ACL attributes using gpfs built-in commands====<br />
Courtesy of Agata Disks (http://csngwinfo.in2p3.fr/mediawiki/index.php/GPFS_ACL)<br />
<br />
<pre><br />
#!/bin/bash<br />
# USAGE<br />
# - on one directory: ./set_acl.sh dir_name<br />
# - on more directories: ./set_acl.sh 'dir_nam*'<br />
#<br />
<br />
# Path of the file that contains the ACL<br />
ACL_FILE_PATH=/agatadisks/data/acl_file.acl<br />
<br />
# Directories onto the ACLs have to be set<br />
dirs=$1<br />
<br />
# Recursive function that sets ACL to files and directories<br />
set_acl () {<br />
curr_dir=$1<br />
for args in $curr_dir/*<br />
do<br />
if [ -f $args ]; then<br />
echo "ACL set on file $args"<br />
mmputacl -i $ACL_FILE_PATH $args<br />
if [ $? -ne 0 ]; then<br />
echo "ERROR: ACL not set on $args"<br />
exit -1<br />
fi<br />
fi<br />
if [ -d $args ]; then<br />
# Set Default ACL in directory<br />
mmputacl -i $ACL_FILE_PATH $args -d<br />
if [ $? -ne 0 ]; then<br />
echo "ERROR: Default ACL not set on $args"<br />
exit -1<br />
fi<br />
echo "Default ACL set on directory $args"<br />
# Set ACL in directory<br />
mmputacl -i $ACL_FILE_PATH $args<br />
if [ $? -ne 0 ]; then<br />
echo "ERROR: ACL not set on $args"<br />
exit -1<br />
fi<br />
echo "ACL set on directory $args"<br />
set_acl $args<br />
fi<br />
done<br />
}<br />
for dir in $dirs<br />
do<br />
if [ ! -d $dir ]; then<br />
echo "ERROR: $dir is not a directory"<br />
exit -1<br />
fi<br />
set_acl $dir<br />
done<br />
exit 0<br />
<br />
</pre><br />
<br />
==[[HPSS|'''High Performance Storage System (HPSS)''']]==<br />
<br />
<br />
==More questions on data management?==<br />
<br />
Check out the [[FAQ#Data_on_SciNet_disks|FAQ]].</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Scheduler&diff=9385Scheduler2018-08-09T17:32:02Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
The queueing system used at SciNet is based around Cluster Resources [http://www.adaptivecomputing.com/products/hpc-products/moab-hpc-basic-edition/ Moab Workload Manager].<br />
Moab is used on both the GPC and TCS however [http://www.adaptivecomputing.com/products/open-source/torque/ Torque] is used as the backend resource manager on the GPC and IBM's [http://www.ibm.com/software/products/en/tivoliworkloadschedulerloadleveler/ LoadLeveler] is used on the TCS.<br />
<br />
This page outlines some of the most common Moab commands with full documentation available from Moab [http://docs.adaptivecomputing.com/mwm/7-2-9/help.htm#a.gcommandoverview.html%3FTocPath%3D4.0%2520Scheduler%2520Commands|_____0 here], the torque (pbs) commands full documentation is [http://docs.adaptivecomputing.com/torque/4-2-9/help.htm here].<br />
<br />
Some common questions about the queuing system can be found on the [[FAQ]] as well.<br />
<br />
=== Queues ===<br />
<br />
==== GPC ====<br />
<br />
===== batch =====<br />
<br />
The batch queue is the default queue on the GPC allowing the user access to all the <br />
resources for jobs between 15 minutes and upto 48 hours. If a specific queue is not specified, <tt>-q</tt> flag,<br />
then a job is submitted to the batch queue. Most jobs will run in batch, and user's can use<br />
feature flags to determine the type of nodes they require.<br />
<br />
For example, to request two nodes anywhere on the GPC (QDR or DDR), use<br />
<pre><br />
#PBS -l nodes=2:ppn=8,walltime=1:00:00<br />
</pre><br />
<br />
For two nodes using DDR, use<br />
<pre><br />
#PBS -l nodes=2:ddr:ppn=8,walltime=1:00:00<br />
</pre><br />
<br />
To get two nodes using QDR, instead, you would use<br />
<pre><br />
#PBS -l nodes=2:qdr:ppn=8,walltime=1:00:00<br />
</pre><br />
<br />
===== debug =====<br />
<br />
A debug queue has been set up primarily for code developers to quickly test<br />
and evaluate their codes and configurations without having to wait in the batch queue. There are 10 nodes<br />
currently reserved for the debug queue. It has quite restrictive limits to promote high turnover<br />
and availability thus a user can only use 2 nodes (16 cores) for 2 hours, to a maximum<br />
of 8 nodes (64 cores) for 1/2 an hour and can only have one job in the debug queue at a time. There<br />
is no minimum time limit on this queue.<br />
<br />
<pre><br />
$ qsub -l nodes=1:ppn=8,walltime=1:00:00 -q debug -I<br />
</pre><br />
<br />
===== largemem =====<br />
<br />
The largemem queue is used for accessing one of two 16 core with 128 GB memory intel Xeon (non-nehalem) nodes. <br />
<br />
<pre><br />
$ qsub -l nodes=1:ppn=16,walltime=1:00:00 -q largemem -I<br />
</pre><br />
<br />
==== TCS ====<br />
<br />
The TCS currently only has one queue, or class, in use called "verylong" and its only<br />
limitation is that jobs must be under 48 hours.<br />
<br />
<pre><br />
#@ class = verylong<br />
</pre><br />
<br />
=== Job Info===<br />
<br />
To see all jobs queued on a system use<br />
<pre><br />
$ showq<br />
</pre><br />
<br />
Three sections are shown; running, idle, and blocked. Idle jobs are commonly referred to as queued jobs <br />
as they meet all the requirements, however they are waiting for available resources. Blocked jobs <br />
are either caused by improper resource requests or more commonly by exceeding a user or groups allowable<br />
resources. For example if you are allowed to submit 10 jobs and you submit 20, the first 10<br />
jobs will be submitted properly and either run right away or be queued, however the other 10 jobs<br />
will be blocked and the jobs won't be submitted to the queue until one of the first 10 finishes.<br />
<br />
=== Available Resources ===<br />
<br />
Determining when your job will run can be tricky as it involves a combination of queue type, node type, system reservations, and job priority. The following commands are provided to help you figure out what resources are currently available, however they may not tell you exactly when your job will run for the aforementioned reasons.<br />
<br />
==== GPC ====<br />
To show how many qdr nodes are currently free, use the show back fill command<br />
<pre><br />
$ showbf -f qdr<br />
</pre><br />
<br />
To show how many ddr nodes are free, use<br />
<pre><br />
$ showbf -f ddr<br />
</pre><br />
<br />
==== TCS ====<br />
To show how many TCS nodes are free, use<br />
<pre><br />
$ showbf -c verylong<br />
</pre><br />
<br />
For example checking for a qdr job<br />
<br />
<pre><br />
$ showbf -f qdr<br />
Partition Tasks Nodes Duration StartOffset StartDate<br />
--------- ----- ----- ------------ ------------ --------------<br />
ALL 14728 1839 7:36:23 00:00:00 00:23:37_09/24<br />
ALL 256 30 INFINITY 00:00:00 00:23:37_09/24<br />
</pre><br />
<br />
shows that for jobs under 7:36:23 you can use 1839 nodes, but if you submit<br />
a job over that time only 30 will be available. In this case this is<br />
due to a large reservation made my SciNet staff, but from a users point<br />
of view, showbf tells you very simply what is available and at what time point.<br />
In this case, a user may wish to set #PBS -l walltime=7:30:00 in their script, or add -l walltime=7:30:00 to their qsub command in order to ensure that the jobs backfill the reserved nodes.<br />
<br />
'''NOTE:''' showbf shows currently available nodes, however just because nodes are available<br />
doesn't mean that your job will start right away. Job priority, system reservations <br />
along with dedicated nodes, such as those for the debug queue, will alter when jobs <br />
run so even if enough nodes appear "free", it doesn't mean your job will actually run right <br />
away.<br />
<br />
=== Job Submission ===<br />
<br />
==== Interactive ====<br />
<br />
On the GPC an interactive queue session can be requested using the following <br />
<pre><br />
$ qsub -l nodes=2:ppn=8,walltime=1:00:00 -I<br />
</pre><br />
<br />
In case you may experience longer than usual delays when requesting an interactive session, try specifying the "debug" queue:<br />
<pre><br />
$ qsub -l nodes=2:ppn=8,walltime=1:00:00 -I -q debug<br />
</pre><br />
<br />
==== Non-interactive (Batch) ====<br />
<br />
For a non-interactive job submission you require a submission script formatted for the appropriate resource manger. Examples<br />
are provided for the [[GPC_Quickstart#Submitting_A_Batch_Job | GPC]] and [[TCS_Quickstart#Submitting_A_Job | TCS]].<br />
<br />
=== Job Status ===<br />
<pre><br />
$ qstat jobid<br />
</pre><br />
To see the status of all your jobs, use<br />
<pre><br />
showq -u username<br />
</pre><br />
or<br />
<pre><br />
qstat -u username<br />
</pre><br />
<br />
If your job appears to be blocked, you can try the following command<br />
<pre><br />
$ checkjob jobid<br />
</pre><br />
which gives more verbose and often a bit confusing output.<br />
<br />
=== Cancel a Job ===<br />
<br />
<pre><br />
$ canceljob jobid<br />
</pre><br />
<br />
To cancel all of your jobs, try this command:<br />
<br />
<pre><br />
$ canceljob `showq | grep yourusername | cut -c -8` <br />
</pre><br />
<br />
or using the torque commands (these tend to work faster for larger number of jobs)<br />
<br />
<pre><br />
$ qdel `qstat | grep yourusername | cut -c -8` <br />
</pre><br />
<br />
=== Accounting ===<br />
<br />
For any user with an NRAC/LRAC allocation, a special account with the Resource Allocation Project (RAP) identifier (RAPI) from Compute Canada Database (CCDB) is set up in order to access the allocated resources. Please use the following instructions to run your job using your special allocation. This is necessary both for accounting purposes as well as to assign the appropriate priority to your jobs.<br />
<br />
Each job run on the system will have a default RAP associated with it. Most users already have their default RAP properly set. However, if you have more than one allocation (different RAPs), you may need/want to change your default RAP in order to charge your jobs to a particular RAP.<br />
<br />
==== Changing your default RAP ====<br />
<br />
# Go to the [https://portal.scinet.utoronto.ca portal], login with your SciNet username and password.<br />
# Click on "Change SciNet default RAP" and change your default RAP.<br />
<br />
==== Specifying the RAP for GPC ====<br />
<br />
Alternatively, you may want to assign a RAP for each particular job you run. There are two ways to specify an account for Moab/Torque: From the command line or inside the batch submission script.<br />
<br />
===== Command line =====<br />
<br />
Use the '-A RAPI' flag when you submit your job using qsub. Note that the command line option will override the submission script if an account is specified on both the submission script and the command line. "RAPI" is the RAP Identifier, e.g. abc-123-de.<br />
<br />
===== Submission Script =====<br />
<br />
Add a line in your submit script as follows:<br />
<pre><br />
#PBS -A RAPI<br />
</pre><br />
<br />
Please replace "RAPI" with your RAP Identifier.<br />
<br />
==== Specifiying the RAP for TCS ====<br />
<br />
Add a line in your submit script as follows:<br />
<pre><br />
# @ account_no = RAPI<br />
</pre><br />
<br />
Please replace "RAPI" with your RAP Identifier.<br />
<br />
<br />
=== User Stats ===<br />
<br />
Show current usage stats for a $USER<br />
<br />
<pre><br />
$ showstats -u $USER<br />
</pre><br />
<br />
=== Reservations ===<br />
<pre><br />
$ showres<br />
</pre><br />
<br />
Standard users can only see their reservations not other users or system ones.<br />
To determine what is available a user can use "showbf", it shows what resources are<br />
available and at what time level, taking into account running jobs and all the reservations. Refer to the [[Moab#Available_Resources | Available Resources]] section of this page for more details.<br />
<br />
=== Job Dependencies ===<br />
<br />
Sometimes you may want one job not to start until another job finishes, however<br />
you would like to submit them both at the same time. This can be done<br />
using job dependencies on both the GPC and TCS, however the commands <br />
are different due to the underlying resource managers being different.<br />
<br />
==== GPC ====<br />
<br />
Use the -W flag with the following syntax in your submission script to have this job not start<br />
until the job with jobid has successfully finished<br />
<pre><br />
-W depend=afterok:jobid<br />
</pre><br />
<br />
This functionality also allows to add dependencies on several jobs, eg.<br />
<br />
<pre><br />
-W depend=afterok:jobid1:jobid2:...:jobidN<br />
</pre><br />
<br />
which in this case, the job will not start until all the jobs jobid1, jobid2, ..., jobidN have successfully finished. Note that the length of the string "afterok:jobid1:jobid2...:jobidN" cannot exceed 1024 characters.<br />
<br />
<br />
More detailed syntax, dependency options and examples can be found <br />
[[http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#-W here]].<br />
<br />
==== TCS ====<br />
<br />
Loadleveler does job dependencies using what they call steps.<br />
See the [[TCS_Quickstart#Steps | TCS Quickstart]] guide for an example.<br />
<br />
=== Adjusting Job Priority ===<br />
<br />
<br />
The ability to adjust job priorities downwards can also be of use to adjust relative priorities of jobs between users who are running jobs of the same allocation (eg, a default or NRAC allocation of the same PI). Priorities are determined by how much of the time of that allocation been currently used, and all users using that account will have identical priorities. This mechanism allows users to voluntarily reduce their priority to allow other users of the same allocation to run ahead of them.<br />
<br />
In principle, by adjusting a jobs priority downwards, you could reduce your jobs priority to the point that someone elses job entirely could go ahead of yours. In practice, however, this is extremely unlikely. Users with NRAC allocations have priorities that are extremely large positive numbers that depend on their allocation and how much of it they have already used during the past fairshare window (2 weeks); it is very unlikely that two groups would have priorities that are within 10 or 100 or 1000 of each other.<br />
<br />
Note that at the moment, we do not allow priorities to go negative; they are integers that can go no lower than 1. (This may change in the future) That means that users of accounts that have already used their full allocation during the current fairshare period (eg, over the past two weeks), and so whose priority would normally be negative but is capped at 1, can not lower their priority any further. Similar, users with a `default' allocation have priority 1, and cannot lower their priorities any further.<br />
<br />
==== GPC ====<br />
<br />
Moab allows users to adjust their jobs' priority moderately downwards, with the <tt>-p</tt> flag; that is, on a qsub line<br />
<pre><br />
$ qsub ... -p -10 JOBID<br />
</pre><br />
or in a script<br />
<pre><br />
...<br />
#PBS -p -10<br />
..<br />
</pre><br />
<br />
The number used (-10 in the examples above) can be any negative number down to -1024. <br />
<br />
The ability to adjust job priorities downwards can be useful when you are running a number of jobs and want some to enter the queue at higher priorities than others. Note that if you absolutely require some jobs to start before others, you could use [[#Job Dependencies | job dependencies]] instead.<br />
<br />
<br />
For a job that is currently queued, one can adjust its priority with<br />
<pre><br />
$ qalter -p -10 JOBID<br />
</pre><br />
<br />
<!--<br />
This is OLD, pre-fairshare info<br />
<br />
==== TCS ====<br />
<br />
TCS users can adjust their priorities by putting the following line in their scripts<br />
<br />
<pre><br />
#@ user_priority = XXX<br />
</pre><br />
<br />
where the number can be between 0 (which is 50 below the default priority) to 1 (the default priority).<br />
--><br />
<br />
=== Suspending a Running Job ===<br />
<br />
Separate from, and in addition to, the ability to place a hold on a queued job, you may want to suspend a running job. For example, you may want to test the timing of events in a weakly coupled parallel environment.<br />
<br />
==== GPC ====<br />
<br />
To suspend a job:<br />
<br />
<pre><br />
qsig -s STOP <jobid><br />
</pre><br />
<br />
and to start it again:<br />
<br />
<pre><br />
qsig -s CONT <jobid>.<br />
</pre><br />
<br />
Scripts are suspendable by default, so you don't need to add any signal handling for this to work.<br />
As far as we can tell, the result is identical to using fg and ctrl-Z (or kill -STOP <PID>) in an interactive run.<br />
<br />
More about using (and trapping) signals can be found on the [[Using Signals]] page.<br />
<br />
<br />
=== QDR Network Switch Affinity ===<br />
<br />
The QDR network is globally 5:1 oversubscribed, but on switch it has full 1:1 cross-section whereas the DDR is completely 1:1 non-blocking. <br />
When a job is submitted to the GPC QDR nodes, the queuing system tries to fulfill the requirements with nodes on the same switch to improve network <br />
performance. If not enough nodes are available on the same switch to satisfy the job request, the queue will then use any available nodes. This behavior can be changed <br />
by using the submission flag " -l nodesetisoptional=false " which forces the queuing system to only<br />
run this job on the same switch, thus the job will stay queued until enough nodes on one switch are available<br />
to satisfy this request. Note that the maximum number of nodes on one switch is 30, so <br />
a request of greater than 30 nodes with this flag will never run. <br />
<pre><br />
qsub -l nodes=4:ppn=8,walltime=10:00,nodesetisoptional=false<br />
</pre><br />
<br />
=== Multiple Job Submissions ===<br />
<br />
If you are doing doing batch processing of a number of similar jobs on the GPC, torque has a feature called job arrays that can be used to simplify this process. <br />
By using the "-t 0-N" option on the command line during job submission or putting it in the job script file, #PBS -t 0-N, torque will expand your <br />
single job submission into N+1 jobs and sets the environment variable PBS_ARRAYID equal to that jobs specific number, ie 0-N, for each job. This <br />
reduces the amount of calls to qsub, and can allow the user to have many less submission scripts. Job arrays also have the benefit of batching <br />
groups of jobs allowing commands like qalter, qdel, qhold to work on all or a subset of the job array jobs with one command, instead of having <br />
to run the command for each job.<br />
<br />
In the following example, 10 jobs are submitted using a single command <br />
<pre><br />
qsub -t 0-9 jobscript.sh<br />
</pre><br />
<br />
and the submission script then modifies the job based on the PBS_ARRAYID.<br />
<pre><br />
#!/bin/bash<br />
#PBS -l nodes=1:ppn=8,walltime=10:00:00<br />
#PBS -N array_jobs<br />
<br />
cd ${PBS_O_WORKDIR}<br />
mkdir job.${PBS_ARRAYID}<br />
cd job.${PBS_ARRAYID}<br />
<br />
echo "Running job ${PBS_ARRAYID}"<br />
mpirun -np 8 ./mycode >& array_job.${PBS_ARRAYID}.out<br />
</pre><br />
<br />
The JOBID and the job name both get the additional ARRAYID added onto them in the form of a hyphen, ie JOBID-ARRAYID. If for example you wanted<br />
to cancel all the jobs in a job array you would use "qdel JOBID", whereas if you wanted to cancel just one of the jobs you would use <br />
"qdel JOBID-ARRAYID". <br />
<br />
See [http://docs.adaptivecomputing.com/torque/4-1-3/Content/topics/2-jobs/multiJobSubmission.htm here] and [http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm here]<br />
for full details.<br />
<br />
<br />
=== Job Tools and Scripts ===<br />
<br />
The following list of commands and scripts are useful at the moment of monitoring and managing submissions and jobs.<br />
We have discussed them in a recent [https://support.scinet.utoronto.ca/education/go.php/183/file_storage/index.php/download/1/files%5B%5D/3702/ TechTalk].<br />
More details can be found [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities here].<br />
<br />
<!-- showq --><br />
<!-- showbf --><br />
<!-- checkjob --><br />
<!-- showstats --><br />
<!--<br />
<pre><br />
showstart jobID<br />
</pre><br />
--><br />
<br />
In addition, the following scripts are available:<br />
<pre><br />
diskUsage<br />
</pre><br />
informs about the user and group file system usage.<br />
<br />
<pre><br />
quota<br />
</pre><br />
offers a shorter version for the user.<br />
<br />
<pre><br />
qsum<br />
</pre><br />
similar to showq, summarizes by user.<br />
<br />
<pre><br />
jobperf jobID<br />
</pre><br />
informs about the performance per-node of a given job.<br />
<br />
<pre><br />
jobError <jobID | jobNAME><br />
</pre><br />
displays on realtime the error output of a given job.<br />
<br />
<pre><br />
jobOutput <jobID | jobNAME><br />
</pre><br />
displays on realtime the standard output of a given job.<br />
<br />
<pre><br />
jobcd <jobID | jobNAME><br />
</pre><br />
allows users to quickly move into the working directory of a given job.<br />
<br />
<pre><br />
jobscript <jobID | jobNAME><br />
</pre><br />
displays the submission script used when submitting a given job.<br />
<br />
<pre><br />
jobssh <jobID | jobNAME><br />
</pre><br />
allows users to connect to the head-node of a given job.<br />
<br />
<pre><br />
jobtop <jobID | jobNAME><br />
</pre><br />
allows users to "top" on the head-node of a given job.<br />
<br />
<pre><br />
jobtree [user]<br />
</pre><br />
displays the jobs tree of dependencies for a given user<br />
<br />
<pre><br />
jobdep <jobID><br />
</pre><br />
displays the dependencies of a given job.<br />
<br />
<pre><br />
jobperf <jobID><br />
</pre><br />
displays current performance statistics of a given job (must be running).<br />
<br />
<br />
===Monitoring Jobs===<br />
[https://wiki.scinet.utoronto.ca/wiki/images/a/a0/TechTalkJobMonitoring.pdf Tech Talk on Monitoring Jobs]<br />
<br />
<br />
===Checking the memory usage from jobs===<br />
<br />
In many occasions it can be really useful to take a look at how much memory your job is using while it is running.<br />
There a couple of ways to do so:<br />
<br />
1) using some of the [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities command line utilities] we have developed, e.g: by using the '''jobperf''' or '''jobtop''' utilities, it will allow you to check the job performance and head's node utilization respectively.<br />
<br />
2) ''ssh'' into the nodes where your job is being run and check for memory usage and system stats right there. For instance, trying the 'top' or 'free' commands, in those nodes.<br />
<br />
Also, it always a good a idea and strongly encouraged to inspect the standard output-log and error-log generated for your job submissions.<br />
These files are named respectively: ''JobName.{o|e}jobIdNumber''; where ''JobName'' is the name you gave to the job (via the '-N' PBS flag) and ''JobIdNumber'' is the id number of the job.<br />
These files are saved in the working directory after the job is finished, but they can be also accessed on real-time using the '''jobError''' and '''jobOutput''' [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities command line utilities].<br />
<br />
Other related topics to memory usage: <br><br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Ram_Disk Using Ram Disk]<br />
<br><br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Memory_Configuration Different Memory Configuration nodes]<br />
<br><br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/FAQ#Monitoring_jobs_in_the_queue Monitoring Jobs in the Queue]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Software_and_Libraries&diff=9384Software and Libraries2018-08-09T17:31:23Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
= Software Module System =<br />
<br />
All the software listed on this page is accessed using a modules system. This means that much of the software is not <br />
accessible by default but has to be loaded using the module command. The<br />
reason is that<br />
* it allows us to easily keep multiple versions of software for different users on the system;<br />
* it allows users to easily switch between versions.<br />
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.<br />
<br />
Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.<br />
<br />
{|<br />
!{{Hl2}}|Function<br />
!{{Hl2}}|Command<br />
!{{Hl2}}|Comments<br />
|-<br />
|List available software packages:<br />
|<pre>$ module avail</pre><br />
|<br />
*If a module is not listed here, it is not supported.<br />
*The flag "(default)" is never part of the name.<br />
|-<br />
|Use particular software:<br />
|<pre> $ module load [module-name] </pre><br />
|<br />
*If possible, specify only the short name (the part before the "/"). <br />
*When ambiguous, this loads the default one. <br />
|-<br />
|List available versions of a specific software package:<br />
|<pre>$ module avail [short-module-name]</pre><br />
|<br />
|-<br />
|List currently loaded modules:<br />
|<pre>$ module list</pre><br />
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.<br />
|-<br />
|Get description of a particular module:<br />
|<pre>$ module help [module-name]</pre><br />
|<br />
|-<br />
|Remove a module from your shell:<br />
|<pre>$ module unload [module-name]</pre><br />
|<br />
|-<br />
|Remove all modules:<br />
|<pre>$ module purge</pre><br />
|<br />
|-<br />
|Replace one loaded module with another:<br />
|<pre>$ module switch [old-module-name] [new-module-name]</pre><br />
|<br />
|-<br />
|Find a module by name and check its dependencies:<br />
|<pre>$ module find [part-of-module-name]</pre><br />
| This is a module extension by SciNet for the GPC<br />
|-<br />
|Find one way to resolve module dependencies:<br />
|<pre>$ module advice [module-name]</pre><br />
| This is a module extension by SciNet for the GPC<br />
|}<br />
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention<br />
<pre><br />
SCINET_[short-module-name]_BASE<br />
SCINET_[short-module-name]_LIB<br />
SCINET_[short-module-name]_INC<br />
</pre><br />
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.<br />
<br />
So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>. <br />
<br />
Errors in loaded modules can arise for a few reasons, for instance:<br />
* A module by that name may not exist.<br />
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.<br />
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.<br />
<br />
It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.<br />
<br />
===Makefile example of using modules===<br />
<br />
For a single-file c++-code using boost's threading library, the Makefile could contain<br />
<br />
<pre><br />
CPPFLAGS=-I${SCINET_BOOST_INC}<br />
LDFLAGS=-L${SCINET_BOOST_LIB}<br />
LDLIBS=-lboost_thread<br />
all: boostthreadexample<br />
boostthreadexample: boostthreadexample.o<br />
boostthreadexample.o: boostthreadexample.cpp<br />
clean:<br />
\rm -f boostthreadexample.o<br />
</pre><br />
<br />
On the command line, you'd type<br />
<br />
<pre><br />
$ module load intel/15.0.2 openmpi/intel/1.6.4 cxxlibraries/boost/1.55.0-intel<br />
$ make<br />
</pre><br />
(Note: openmpi is loaded here because the boost module includes the boost:mpi package, which is however not used in this example).<br />
<br />
===CMake example of using modules===<br />
<br />
Several versions of CMake are installed on the GPC: the default version is 2.6 (which may be too old in many cases), but there are more version available as modules Currently (Dec 2015), the most recent installed version is <tt>cmake/3.4.0</tt> (type <tt>module avail cmake</tt> to see all available version.<br />
<br />
Using cmake with the environment module eco-system requires a few tweaks. Because these tweaks are almost never discussed in the compilation instructions of package that use cmake, and are even hard to find in the CMake documentation itself, we explain these tweaks here.<br />
<br />
For compiling the above example of a single-file c++ code using boost's threading library with CMake, one way is to have the CMakeLists.txt could contain<br />
<br />
<pre><br />
include_directories($ENV{SCINET_BOOST_INC})<br />
link_directories($ENV{SCINET_BOOST_LIB})<br />
cmake_minimum_required(VERSION 2.6)<br />
project(boostthreadexample)<br />
add_executable(boostthreadexample boostthreadexample.cpp)<br />
target_link_libraries(boostthreadexample boost_thread)<br />
</pre><br />
<br />
Make sure the include_directories and link_directories commands are given before the add_executable, or they may not take effect.<br />
<br />
On the command line, you could then compile as follows:<br />
<br />
<pre><br />
$ module load intel/15.0.2 openmpi/intel/1.6.4 cxxlibraries/boost/1.55.0-intel<br />
$ cmake .<br />
$ make<br />
</pre><br />
Although it is better to use a separate build directory. (Note: openmpi is loaded here because the boost module includes the boost:mpi package, which is however not used in this example).<br />
<br />
An <i>alternative setup</i> for working cmake is to keep the CMakeLists.txt simple and standard:<br />
<pre><br />
cmake_minimum_required(VERSION 2.6)<br />
project(boostthreadexample)<br />
add_executable(boostthreadexample boostthreadexample.cpp)<br />
target_link_libraries(boostthreadexample boost_thread)<br />
</pre><br />
And then to use the environment variables to get the paths<br />
<pre><br />
$ module load intel/15.0.2 openmpi/intel/1.6.4 cxxlibraries/boost/1.55.0-intel<br />
$ CMAKE_PREFIX=${SCINET_BOOST_BASE}<br />
$ CMAKE_INCLUDE_PATH=${SCINET_BOOST_INC}<br />
$ CMAKE_LIBRARY_PATH=${SCINET_BOOST_LIB}<br />
$ CMAKE_PROGRAM_PATH=${SCINET_BOOST_BIN}<br />
$ cmake .<br />
$ make<br />
</pre><br />
<br />
== Default and non-default modules ==<br />
<br />
When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.<br />
<br />
== Naming convention ==<br />
<br />
For modules that access applications, the full name of a module is as follows.<br />
<pre><br />
[short-module-name]/[version-number]<br />
</pre><br />
<br />
To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:<br />
{|border="1" cellspacing="0" cellpadding="2"<br />
| '''old name'''<br />
| '''new name'''<br />
| '''remarks'''<br />
|-<br />
|autoconf/autoconf-2.64 &nbsp; &nbsp;&nbsp;<br />
|autoconf/2.64<br />
|''short name unchanged''<br />
|-<br />
|cuda/cuda-3.0 <br />
|cuda/3.0<br />
|''default's short name unchanged''<br />
|-<br />
|cuda/cuda-3.1 <br />
|cuda/3.1<br />
|<br />
|-<br />
|debuggers/ddd-3.3.12 <br />
|ddd/3.3.12<br />
|<br />
|-<br />
|debuggers/gdb-7.1 <br />
|gdb/7.1<br />
|<br />
|-<br />
|editors/nano/2.2.4 <br />
|nano/2.2.4<br />
|<br />
|-<br />
|emacs/emacs-23.1 <br />
|emacs/23.1.1<br />
|''short name unchanged''<br />
|-<br />
|gcc/gcc-4.4.0 <br />
|gcc/4.4.0<br />
|''short name unchanged''<br />
|-<br />
|graphics/ncview <br />
|ncview/1.93<br />
|<br />
|-<br />
|graphics/graphics <br />
|grace/5.1.22<br />
|<br />
|-<br />
| <br />
|gnuplot/4.2.6<br />
|<br />
|-<br />
|svn/svn165 <br />
|svn/1.6.5<br />
|''short name unchanged''<br />
|-<br />
|visualization/paraview <br />
|paraview/3.8<br />
|<br />
|-<br />
|amber10/amber10 <br />
|amber/10.0.30 <br />
|<br />
|-<br />
|gamess/gamess <br />
|gamess/May2209 &nbsp;<br />
|''default's short name unchanged''<br />
|}<br />
<br />
==modulefind - Finding modules by name==<br />
<br />
The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.<br />
<br />
A little SciNet utility called <tt>modulefind</tt> (one word) or <tt>module find</tt> (two words) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have <br />
been loaded, could be loaded, cannot because of conflicts with<br />
already loaded modules, or have unresolved dependencies <br />
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.<br />
<br />
Note that just <tt>module find</tt> will list all top-level modules.<br />
<br />
== Making your own modules ==<br />
<br />
How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.<br />
<br />
== Deprecated modules ==<br />
<br />
Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre><br />
<br />
<!--<br />
<br />
Currently (Oct 5,2010), the following modules are deprecated on the GPC: <br />
<pre><br />
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed<br />
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig<br />
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old <br />
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old<br />
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025 <br />
</pre><br />
<br />
On the TCS, currently (Oct 5,2010) the only deprecated module is:<br />
<pre><br />
ncl/5.1.1old<br />
</pre><br />
<br />
--><br />
Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.<br />
<br />
== Commercial software ==<br />
<br />
Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc. <br />
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].<br />
<br />
== Other software and libraries ==<br />
<br />
If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.<br />
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which<br />
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.<br />
<br />
If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.<br />
<br />
== Software lists ==<br />
=== Gravity.Viz GPU Software ===<br />
<br />
The CPUs in the GPU nodes of the Gravity cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.<br />
<br />
=== GPC Software ===<br />
<br />
{| border="1" cellpadding="10" cellspacing="0"<br />
!{{Hl2}}| Software <br />
!{{Hl2}}| Versions<br />
!{{Hl2}}| Comments<br />
!{{Hl2}}| Command/Library<br />
!{{Hl2}}| Module Name<br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''<br />
|- <br />
|Intel Compiler<br />
|12.1.3*, 12.1.5, 13.1.1, 14.0.1, 15.0, 15.0.1, 15.0.2, 15.0.6, 16.0.3<br />
| includes MKL library, which includes BLAS, LAPACK, FFT, ...<br />
| <tt>icpc,icc,ifort</tt><br />
| <tt>intel</tt><br />
|- <br />
| GCC Compiler<br />
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2, 4.8.1, 4.9.0, 5.2.0<br />
| Version 5.2.0 supports [[Co-array_Fortran_on_the_GPC|coarrays]]<br />
| <tt>gcc,g++,gfortran<br>caf,cafrun (gcc/5.2.0 only)</tt><br />
| <tt>gcc</tt><br />
|-<br />
| Cuda<br />
| 3.2, 4.0, 4.1*, 4.2, 5.0, 5.5, 6.0<br />
| NVIDIA's extension to C for GPGPU programming<br />
| <tt>nvcc</tt><br />
| <tt>cuda</tt><br />
|- <br />
| PGI Compiler<br />
| 13.2<br />
| supports OpenACC and CUDA Fortran <br />
| <tt>pgcc,pgcpp,pgfortran</tt><br />
| <tt>pgi</tt><br />
|-<br />
| IntelMPI<br />
| 4.0.2, 4.0.3, 4.1.2*, 5.0.1, 5.0.2<br />
| MPICH2 based MPI (intelmpi/5.* are in the 'experimental' modules)<br />
| <tt>mpicc,mpiCC,mpif77,mpif90</tt><br />
| <tt>intelmpi</tt><br />
|- <br />
| OpenMPI<br />
| 1.4.4*, 1.5.4, 1.6.4, 1.8.3<br />
|<br />
| <tt>mpicc,mpiCC,mpif77,mpif90</tt><br />
| <tt>openmpi</tt><br />
|-<br />
| UPC<br />
| 2.12.2<br />
| Berkley Unified Parallel C Implementation<br />
| <tt>upcc</tt><br />
| <tt>upc</tt><br />
|-<br />
| Haskell<br />
| 7.4.2*, 6.12.3<br />
| Haskell compiler and Cabal package manager<br />
| <tt>ghc,cabal</tt><br />
| <tt>haskell</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''<br />
|- <br />
| Nano<br />
| 2.2.4<br />
| Nano's another editor<br />
| <tt>nano</tt><br />
| <tt>nano</tt><br />
|- <br />
| Emacs<br />
| 23.1.1, 24.4*<br />
| New version of popular text editor<br />
| <tt>emacs</tt><br />
| <tt>emacs</tt><br />
|-<br />
| Vim<br />
| 7.2.4*, 7.4.5<br />
| Vi editor. No module needs to be loaded for the default version<br />
| <tt>vim</tt><br />
| <tt>vim</tt><br />
|- <br />
| XEmacs<br />
| 21.4.22<br />
| XEmacs editor<br />
| <tt>xemacs</tt><br />
| <tt>xemacs</tt><br />
|-<br />
| Midnight Commander<br />
| 4.7.0*, 4.8.14<br />
| A visual file manager within the terminal with builtin editor and viewer<br />
| <tt>mc, mcedit, mcview</tt><br />
| <tt>mc</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''<br />
|-<br />
| Autoconf<br />
| 2.68<br />
|<br />
| <tt>autoconf, ...</tt><br />
| <tt>autoconf</tt><br />
|- <br />
| Automake<br />
| 1.11.2<br />
|<br />
| <tt>aclocal, automake</tt><br />
| <tt>automake</tt><br />
|- <br />
| CMake<br />
| 2.8.6, 2.8.8*, 3.1.0<br />
| cross-platform, open-source build system<br />
| <tt>cmake</tt><br />
| <tt>cmake</tt><br />
|-<br />
| Scons<br />
| 2.0<br />
| Software construction tool<br />
| <tt>scons</tt><br />
| <tt>scons</tt><br />
|- <br />
| Git<br />
| 1.7.1, 1.7.10*, 1.9.5<br />
| Revision control system<br />
| <tt>git,gitk</tt><br />
| <tt>git</tt><br />
|- <br />
| Git-annex<br />
| 4.20130827, 5.20150219*<br />
| Revision control system<br />
| <tt>git,gitk</tt><br />
| <tt>git-annex</tt><br />
|-<br />
| Intel tools<br />
| 2013, 2015<br />
| Intel Code Analysis Tools<br />
| Vtune Amplifier XE, Inspector XE<br />
| <tt>inteltools</tt><br />
|-<br />
| Mercurial<br />
| 1.8.2<br />
| Version control system<br>(part of the python module!)<br />
| <tt>hg</tt><br />
| <tt>python</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''<br />
|-<br />
| [http://www.allinea.com/products/ddt/ DDT]<br />
| 4.0, 4.1, 4.2.1*<br />
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler<br />
| <tt>ddt, map</tt><br />
| <tt>ddt</tt><br />
|- <br />
| DDD<br />
| 3.3.12<br />
| Data Display Debugger<br />
| <tt>ddd</tt><br />
| <tt>ddd</tt><br />
|- <br />
| GDB<br />
| 7.3.1*, 7.6<br />
| GNU debugger (the intel idbc debugger is available by default)<br />
| <tt>gdb</tt><br />
| <tt>gdb</tt><br />
|- <br />
| MPE2<br />
| 2.4.5<br />
| Multi-Processing Environment with intel + OpenMPI<br />
| <tt>mpecc, mpefc, jumpshot</tt><br />
| <tt>mpe</tt><br />
|-<br />
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]<br />
| 2.1<br />
| sampling and MPI tracing<br />
| <tt>openss, ...</tt><br />
| <tt>openspeedshop</tt><br />
|-<br />
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]<br />
| 1.3.3, 1.4.3*<br />
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)<br />
| <tt>scalasca</tt><br />
| <tt>scalasca</tt><br />
|-<br />
| [http://ipm-hpc.sourceforge.net IPM]<br />
| 0.983<br />
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]<br />
| <tt>ipm, ipm_parse, ploticus,...</tt><br />
| <tt>ipm</tt><br />
|- <br />
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]<br />
| 3.7,0*, 3.9.0<br />
| Memory checking utility<br />
| <tt>valgrind,cachegrind</tt><br />
| <tt>valgrind</tt><br />
|-<br />
| Padb<br />
| 3.2 <br />
| examine and debug parallel programs<br />
| <tt>padb</tt><br />
| <tt>padb</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|<span id="anchor_viz"></span>'''''Visualization tools'''''<br />
|- <br />
| Grace<br />
| 5.1.22<br />
| Plotting utility<br />
| <tt>xmgrace</tt><br />
| <tt>grace</tt><br />
|- <br />
| Gnuplot<br />
| 4.2.6, 4.6.1*<br />
| Plotting utility<br>Requires 'extras' module if used on compute nodes.<br />
| <tt>gnuplot</tt><br />
| <tt>gnuplot</tt><br />
|- <br />
| [[ Using_Paraview | ParaView ]]<br />
| 3.12.0*, 3.14.1, 4.1.0<br />
| Scientific visualization, server only<br />
| <tt>pvserver,pvbatch,pvpython</tt><br />
| <tt>paraview</tt><br />
|- <br />
| VMD<br />
| 1.8.6, 1.9*<br />
| Visualization and analysis utility<br />
| <tt>vmd</tt><br />
| <tt>vmd</tt><br />
|-<br />
| [https://visit.llnl.gov/ VisIt]<br />
| 2.6.3, 2.6.3-parallel*, 2.10.0-bin, 2.10.0-bin_mesa<br />
| Interactive, scalable, visualization, animation and analysis tool<br />
| <tt>visit</tt><br />
| <tt>visit</tt><br />
|- <br />
| NCL/NCARG<br />
| 6.0.0<br />
| NCARG graphics and ncl utilities<br />
| <tt>ncl</tt><br />
| <tt>ncl</tt><br />
|- <br />
| ROOT<br />
| 5.30.00, 5.30.03*, 5.34.03, 6.02.02<br />
| ROOT Analysis Framework from CERN<br />
| <tt>root</tt><br />
| <tt>ROOT</tt> <br />
|-<br />
| ImageMagick<br />
| 6.6.7<br />
| Image manipulation tools<br />
| <tt>convert,animate,composite,...</tt><br />
| <tt>ImageMagick</tt><br />
|-<br />
| PGPLOT<br />
| 5.2.2<br />
| Graphics subroutine library<br />
| <tt>libcpgplot,libpgplot,libtkpgplot</tt><br />
| <tt>pgplot</tt><br />
|-<br />
| [http://users.monash.edu.au/~dprice/splash Splash ]<br />
| 2.6.0<br />
| Visualization suite for SPH simulations<br />
| <tt>asplash, dsplash, gsplash, nsplash, rsplash, srsplash, ssplash, tsplash, vsplash</tt><br />
| <tt>splash</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''<br />
|- <br />
| NetCDF<br />
| 4.1.3*, 4.2.1, 4.3.2<br />
| Scientific data storage and retrieval<br />
| <tt>ncdump,ncgen,libnetcdf</tt><br />
| <tt>netcdf</tt><br />
|- <br />
| Ncview<br />
| 2.1.1, 2.1.<br />
| Visualization for NetCDF files<br />
| <tt>ncview</tt><br />
| <tt>ncview</tt><br />
|- <br />
| NCO<br />
| 4.0.8, 4.3.2<br />
| NCO utilities to manipulate netCDF files<br />
| <tt>ncap, ncap2, ncatted, etc.</tt><br />
| <tt>nco</tt><br />
|- <br />
| CDO<br />
| 1.5.1, 1.5.4*, 1.6.1<br />
| Climate Data Operators<br />
| <tt>cdo</tt><br />
| <tt>cdo</tt><br />
|- <br />
| UDUNITS<br />
| 2.1.11<br />
| unit conversion utilities<br />
| <tt>libudunits2</tt><br />
| <tt>udunits</tt><br />
|- <br />
| HDF4<br />
| 4.2.6<br />
| Scientific data storage and retrieval<br />
| <tt>h4fc,hdiff,...,libdf,libsz</tt><br />
| <tt>hdf4</tt><br />
|- <br />
| [[Hdf5 | HDF5]]<br />
| 1.8.7-v18*<br />
| Scientific data storage and retrieval, parallel I/O<br />
| <tt>h5ls, h5diff, ..., libhdf5</tt><br />
| <tt>hdf5</tt><br />
|- <br />
| [http://www.arg0.net/encfs EncFS ]<br />
| 1.74<br />
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)<br />
| <tt>encfs</tt><br />
| <tt>encfs</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''<br />
|- <br />
| [[amber|AMBER 10]]<br />
| Amber 10 + Amber Tools 1.3<br />
| Amber Molecular Dynamics Package<br />
| <tt>sander, sander.MPI</tt><br />
| <tt>amber</tt><br />
|-<br />
| antlr<br />
| 2.7.7<br />
| ANother Tool for Language Recognition<br />
| <tt>antlr, antlr-config<br>libantlr, antlr.jar, antlr.py</tt><br />
| <tt>antlr</tt><br />
|-<br />
| [[gamess|GAMESS (US)]]<br />
| August 18, 2011 R1<br />
| General Atomic and Molecular Electronic Structure System<br />
| <tt>rungms</tt><br />
| <tt>gamess</tt><br />
|-<br />
| [[gromacs|GROMACS]]<br />
| 4.5.5, 4.5.7, 4.6.2, 4.6.3, 4.6.7, 5.0.4<br />
| GROMACS molecular dynamics, single precision, MPI<br />
| <tt>grompp, mdrun, gmx, gmx_mpi</tt><br />
| <tt>gromacs</tt><br />
|-<br />
| [[namd|NAMD]]<br />
| 2.8, 2.9*<br />
| NAMD - Scalable Molecular Dynamics<br />
| <tt>namdmpiexec, namd2</tt><br />
| <tt>namd</tt><br />
|-<br />
| [[nwchem|NWChem]]<br />
| 6.0<br />
| NWChem Quantum Chemistry<br />
| <tt>nwchem</tt><br />
| <tt>nwchem</tt><br />
|-<br />
| [http://www.quantum-espresso.org/index.php Quantum Espresso]<br />
| 4.3.2, 5.0.3<br />
| Quantum Chemistry <br />
| <tt>pw.x, ...</tt><br />
| <tt>espresso</tt><br />
|-<br />
| [http://blast.ncbi.nlm.nih.gov BLAST]<br />
| 2.2.23+<br />
| Basic Local Alignment Search Tool<br />
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt><br />
| <tt>blast</tt><br />
|-<br />
|-<br />
| [http://denovoassembler.sourceforge.net RAY]<br />
| 2.1.0 (small k-mer), 2.2.0, 2.3.1<br />
| Parallel de novo genome assemblies<br />
| <tt>Ray</tt><br />
| <tt>ray</tt><br />
|-<br />
| [[https://www.cp2k.org CP2K]]<br />
| 3.0<br />
| DFT molecular dynamics, MPI<br />
| <tt>cp2k.psmp</tt><br />
| <tt>cp2k</tt><br />
|-<br />
| [[cpmd|CPMD]]<br />
| 3.13.2<br />
| Carr-Parinello molecular dynamics, MPI<br />
| <tt>cpmd.x</tt><br />
| <tt>cpmd</tt><br />
|-<br />
| [[R Statistical Package|R]] <br />
| 2.13.1, 2.14.1, 2.15.1*, 3.0.0, 3.0.1, 3.1.1<br />
| statistical computing<br />
| <tt>R</tt><br />
| <tt>R</tt><br />
|-<br />
| Octave<br />
| 3.4.3*, 3.8.1<br />
| Matlab-like environment<br />
| <tt>octave</tt><br />
| <tt>octave</tt><br />
|-<br />
| [http://www.openfoam.org OpenFOAM ]<br />
| 2.1.0, 2.3.0<br />
| Open Source CFD Package<br />
| <tt>*foam</tt><br />
| <tt>openfoam</tt><br />
|-<br />
| Bedtools<br />
| 2.21.0<br />
| Toolset for genome arithmetic<br />
| <tt>bedtools ...</tt><br />
| <tt>bedtools</tt><br />
|-<br />
| Samtools<br />
| 0.1.19<br />
| Suite of programs for interacting with high-throughput sequencing data<br />
|<br />
| <tt>samtools</tt><br />
|-<br />
| [http://mrbayes.sourceforge.net/ MrBayes ]<br />
| 3.2.4<br />
| MrBayes is a program for Bayesian inference and model choice across a wide range of phylogenetic and evolutionary models.<br />
| <tt>mb</tt><br />
| <tt>mrbayes</tt><br />
|-<br />
| Stacks<br />
| 1.28 1.29*<br />
| A software pipeline for building loci from short-read sequences.<br />
| <br />
| <tt>stacks</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''<br />
|- <br />
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]<br />
| 3.1*, 3.2, 3.3, 3.4.4<br />
| Portable, Extensible Toolkit for Scientific Computation (PETSc)<br />
| <tt>libpetsc, etc.. </tt><br />
| <tt>petsc</tt><br />
|-<br />
| BOOST<br />
| 1.47.0, 1.54, 1.55<br />
| C++ Boost libraries<br />
| <tt>libboost...</tt><br />
| <tt>cxxlibraries/boost</tt><br />
|-<br />
| Rarray<br />
| 1.0<br />
| C++ multidimensional array library<br />
| <tt>rarray</tt><br />
| <tt>cxxlibraries/rarray</tt><br />
|- <br />
| [[Armadillo C++ linear algebra library | Armadillo]]<br />
| 3.910.0<br />
| C++ armadillo libraries (implement Matlab-like syntax)<br />
| <tt>armadillo</tt><br />
| <tt>armadillo</tt><br />
|- <br />
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]<br />
| 1.13<br />
| Optimized BLAS implementation <br />
| <tt>libgoto2</tt><br />
| <tt>gotoblas</tt><br />
|- <br />
| OpenBLAS<br />
| 1.13, 0.2.13*<br />
| Open BLAS implementation including lapack, cblas and lapacke. Note that compilations with the Intel c/c++ compiler will need to link with <tt>-lifcore</tt><br />
| <tt>libopenblas</tt><br />
| <tt>openblas</tt><br />
|- <br />
| GSL<br />
| 1.13*, 1.15<br />
| GNU Scientific Library<br />
| <tt>libgsl, libgslcblas</tt><br />
| <tt>gsl</tt><br />
|- <br />
| FFTW<br />
| 2.1.5, 3.3.0, 3.3.3*<br />
| fast Fourier transform library<br />
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL<br />
| <tt>libfftw3</tt><br />
| <tt>fftw</tt><br />
|- <br />
| LAPACK<br />
| <br />
| Provided by the Intel MKL library<br />
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/<br />
| <tt>intel</tt><br />
|- <br />
| [http://freshmeat.net/projects/rlog RLog ]<br />
| 1.4<br />
| RLog provides a flexible message logging facility for C++ programs and libraries.<br />
| <tt>librlog</tt><br />
| <tt>cxxlibraries/rlog</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''<br />
|-<br />
| [[GNU Parallel]]<br />
| 2012-10-22, 20140622*<br />
| execute commands in parallel<br />
| <tt>parallel</tt><br />
| <tt>gnu-parallel</tt><br />
|- <br />
| [[Python]]<br />
| 2.7.2*, 2.7.3, , 2.7.5, 2.7.8<br />
| Python programming language. See [[Python]] page for details on installed packages.<br />
| <tt>python</tt><br />
| <tt>python</tt><br />
|- <br />
| [[Python]]<br />
| 3.3.4<br />
| Python programming language. Modules included : numpy 1.8.1 , scipy 0.14.0 , matlotlib 1.3.1 , ipython 1.2.1 , cython 0.20.1 , h5py-2.3.0 , tables-3.1.1 , netCDF4-1.1.0 , astropy-0.3.2 , scikit_learn-0.15.0b1<br />
| <tt>python</tt><br />
| <tt>python</tt><br />
|- <br />
| Ruby<br />
| 1.9.1*, 1.9.3<br />
| Ruby programming language<br />
| <tt>ruby</tt><br />
| <tt>ruby</tt><br />
|-<br />
| Java<br />
| 1.6.0, 1.7.1*<br />
| IBM's Java JRE ad SDK<br />
| <tt>java</tt>, <tt>javac</tt><br />
| <tt>java</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Other'''''<br />
|-<br />
| [[VNC]]<br />
|<br />
| 'Virtual Network Computing', an alterative for X forwarding. Only works on the devel nodes. See our [[VNC]] wiki page.<br />
| <tt>vncstart, vncstop, vncstatus,...</tt><br />
| <tt> vnc</tt><br />
|-<br />
| Xlibraries<br />
|<br />
| A collection of X graphics libraries and tools<br />
| <tt>xterm</tt>, <tt>xpdf</tt>, ...<br />
| <tt>Xlibraries</tt><br />
|-<br />
| Extras<br />
|<br />
| A collection of standard linux and home-grown tools<br />
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...<br />
| <tt>extras</tt><br />
|}<br />
<br />
''* Several versions of this module are installed; listed is the default version.''<br />
<br />
<!--<br />
<br />
=== TCS Software ===<br />
{| border="1" cellpadding="10" cellspacing="0"<br />
!{{Hl2}} |Software <br />
!{{Hl2}}| Version<br />
!{{Hl2}}| Comments<br />
!{{Hl2}}| Command/Library<br />
!{{Hl2}}| Module Name<br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''<br />
|-<br />
|IBM compilers<br />
|10.1,11.1,12.1(c/c++)<br>12.1,13.1,14.1(fortran)<br />
| See [[TCS Quickstart]]<br />
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt><br />
| ''oldest versions are standard available'' ''xlf'' ''vacpp''<br />
|-<br />
|IBM MPI library<br />
|<br />
| See [[TCS Quickstart]]<br />
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt><br />
| ''standard available''<br />
|-<br />
| UPC<br />
| 1.2<br />
| Unified Parallel C<br />
| <tt>xlupc</tt><br />
| <tt>upc</tt><br />
|-<br />
|IBM fortran compiler<br />
|13.1, 14.1<br />
| newer version <br />
| <tt>xlf,xlf_r</tt><br />
| xlf/13.1<br />
|-<br />
|IBM c/c++ compilers<br />
|11.1, 12.1<br />
| new versions<br />
| <tt>xlc,xlC,xlc_r,xlC_r</tt><br />
| vacpp<br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''<br />
|- <br />
| [http://www.allinea.com/products/ddt/ DDT]<br />
| 4.0, 4.1, 4.2.1*<br />
| Allinea's Distributed Debugging Tool<br />
| <tt>ddt</tt><br />
| <tt>ddt</tt><br />
|-<br />
| MPE2<br />
| 1.0.6<br />
| Performance Visualization for Parallel Programs <br />
| <tt>libmpe</tt><br />
| <tt>mpe</tt><br />
|-<br />
| Scalasca<br />
| 1.2<br />
| SCalable performance Analysis of LArge SCale Applications<br />
| <tt>scalasca, ...</tt><br />
| <tt>scalasca</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''<br />
|-<br />
| HDF4<br />
| 4.2.5<br />
| Scientific data storage and retrieval<br />
| <tt>h4fc, hdiff, ..., libdf, libsz</tt><br />
| <tt>hdf4</tt><br />
|-<br />
| HDF5<br />
|<br />
| Scientific data storage and retrieval, parallel I/O<br>Part of the extras module on the tcs:<br>compile with <tt>-I${SCINET_EXTRAS_INC}</tt><br> link with <tt>-L${SCINET_EXTRAS_LIB}</tt><br />
| <tt>libhdf5</tt><br />
| <tt>extras</tt><br />
|-<br />
| NetCDF + ncview<br />
| 4.0.1*<br />
| Scientific data storage and retrieval<br />
| <tt>ncdump, ncgen, libnetcdf, ncview</tt><br />
| <tt>netcdf</tt><br />
|-<br />
| parallel netCDF<br />
| 1.1.1*<br />
| Scientific data storage and retrieval using MPI-IO<br />
| <tt>libpnetcdf.a</tt><br />
| <tt>parallel-netcdf</tt><br />
|-<br />
| NCO<br />
| 3.9.6*<br />
| NCO utilities to manipulate netCDF files<br />
| <tt>ncap, ncap2, ncatted, </tt> etc.<br />
| <tt>nco</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''<br />
|-<br />
| FFTW<br />
| 3.2.2<br />
| Fast Fourier transform library<br>Part of the extras module on the tcs:<br>compile with <tt>-I${SCINET_EXTRAS_INC}</tt><br> link with <tt>-L${SCINET_EXTRAS_LIB}</tt><br />
| <tt>libfftw, libfftw_mpi,libfftw3</tt><br />
| <tt>extras</tt><br />
|-<br />
| GSL<br />
| 1.13<br />
| GNU Scientific Library<br />
| <tt>libgsl, libgslcblas</tt><br />
| <tt>gsl</tt><br />
|-<br />
| LAPACK+SCALAPACK<br />
| 3.4.2+2.0.2<br />
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.<br />
| <tt>liblapack,libscalapack,libblacs</tt><br />
| <tt>lapack</tt><br />
|-<br />
| PetSc<br />
| 3.2<br />
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.<br />
| <tt>libpetsc,...</tt><br />
| <tt>petsc</tt><br />
|-<br />
| extras<br />
|<br />
| Adds paths to a fuller set of libraries to your user environment<br> compile with <tt>-I${SCINET_EXTRAS_INC}</tt><br> link with <tt>-L${SCINET_EXTRAS_LIB}</tt><br />
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt><br />
| <tt>extras</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''<br />
|-<br />
| [http://www.quantum-espresso.org/index.php Quantum Espresso]<br />
| 5.4.0<br />
| Quantum Chemistry <br />
| <tt>pw.x, ...</tt><br />
| <tt>espresso</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Other'''''<br />
|-<br />
| gmake<br />
| 3.82<br />
| GNU's make. Replaces AIX make or gmake 3.80.<br />
| <tt>make</tt>, <tt>gmake</tt><br />
| <tt>gmake</tt><br />
|-<br />
| antlr<br />
| 2.7.7<br />
| ANother Tool for Language Recognition<br />
| <tt>antlr, antlr-config<br>libantlr, antlr.jar, antlr.py</tt><br />
| <tt>antlr</tt><br />
|-<br />
| NCL<br />
| 5.1.1<br />
| NCAR Command Language<br />
| <tt>ncl, libncl, ...</tt><br />
| <tt>ncl</tt><br />
|-<br />
|}<br />
''* Several versions of this module are installed; listed is the default version.''<br />
<br />
--><br />
<br />
=== P7 Software ===<br />
{| border="1" cellpadding="10" cellspacing="0"<br />
!{{Hl2}} |Software <br />
!{{Hl2}}| Version<br />
!{{Hl2}}| Comments<br />
!{{Hl2}}| Command/Library<br />
!{{Hl2}}| Module Name<br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''<br />
|-<br />
|IBM fortran compiler<br />
|14.1, 13.1<br />
|See [[P7 Linux Cluster]]<br />
|<tt>xlf,xlf_r,xlf90,...</tt><br />
|xlf<br />
|-<br />
|IBM c/c++ compilers<br />
|12.1, 11.1<br />
|See [[P7 Linux Cluster]]<br />
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt><br />
|vacpp<br />
|-<br />
| Binutils<br />
| 2.23.2<br />
| <br />
| <tt>addr2line, ar, ld, ...</tt><br />
| <tt>binutils</tt><br />
|- <br />
|IBM MPI library<br />
|5.2.2<br />
|IBM's Parallel Environment<br />
|<tt>mpcc,mpCC,mpfort,mpiexec</tt><br />
|pe<br />
|- <br />
| GCC Compiler<br />
| 4.6.1 , 4.8.1<br />
| GNU Compiler Collection<br />
| <tt>gcc,g++,gfortran</tt><br />
| <tt>gcc</tt><br />
|-<br />
| Java<br />
| 7.0<br />
| IBM Java 1.7 implementation<br />
| <tt>javac</tt><br />
| <tt>jdk</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''<br />
|-<br />
| [http://www.allinea.com/products/ddt/ DDT]<br />
| 4.0, 4.1, 4.2.1*<br />
| Allinea's Distributed Debugging Tool<br />
| <tt>ddt</tt><br />
| <tt>ddt</tt><br />
|- <br />
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''<br />
|-<br />
| HDF5<br />
| 1.8.7<br />
| Scientific data storage and retrieval, parallel I/O<br />
| <tt>libhdf5</tt><br />
| <tt>hdf5</tt><br />
|-<br />
| NetCDF<br />
| 4.1.3<br />
| Scientific data storage and retrieval<br />
| <tt>ncdump, ncgen, libnetcdf</tt><br />
| <tt>netcdf</tt><br />
|-<br />
| parallel netCDF<br />
| 1.2.0<br />
| Scientific data storage and retrieval using MPI-IO<br />
| <tt>libpnetcdf.a</tt><br />
| <tt>parallel-netcdf</tt><br />
|-<br />
| NCO<br />
| 4.0.8<br />
| NCO utilities to manipulate netCDF files<br />
| <tt>ncap2, ncatted, </tt> etc.<br />
| <tt>nco</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''<br />
|-<br />
| GSL<br />
| 1.13<br />
| GNU Scientific Library<br />
| <tt>libgsl, libgslcblas</tt><br />
| <tt>gsl</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''<br />
|-<br />
| [[Python]]<br />
| 2.7.5<br />
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1<br />
| <tt>python</tt><br />
| <tt>python</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Other'''''<br />
|-<br />
| gnuplot<br />
| 4.6.1<br />
| command-driven interactive function and data plotting program<br />
| <tt>gnuplot</tt><br />
| <tt>gnuplot</tt><br />
|-<br />
| antlr<br />
| 2.7.7<br />
| ANother Tool for Language Recognition<br />
| <tt>antlr, antlr-config<br>libantlr, antlr.jar, antlr.py</tt><br />
| <tt>antlr</tt><br />
|-<br />
| udunits<br />
| 2.1.11<br />
| unit conversion utilities<br />
| <tt>libudunits2</tt><br />
| <tt>udunits</tt><br />
|-<br />
| extras<br />
|<br />
| Adds paths to a fuller set of applications and libraries to your user environment<br />
| <tt>bindlaunch, ...</tt><br />
| <tt>extras</tt><br />
|}<br />
<br />
=Manuals=<br />
{{:Manuals}}</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=FAQ&diff=9383FAQ2018-08-09T17:30:57Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
__TOC__<br />
<br />
<br />
==The Basics==<br />
===Whom do I contact for support?===<br />
<br />
Whom do I contact if I have problems or questions about how to use the SciNet systems?<br />
<br />
'''Answer:'''<br />
<br />
E-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] <br />
<br />
In your email, please include the following information:<br />
<br />
* your username on SciNet<br />
* the cluster that your question pertains to (GPC or TCS; SciNet is not a cluster!),<br />
* any relevant error messages<br />
* the commands you typed before the errors occured<br />
* the path to your code (if applicable)<br />
* the location of the job scripts (if applicable)<br />
* the directory from which it was submitted (if applicable)<br />
* a description of what it is supposed to do (if applicable)<br />
* if your problem is about connecting to SciNet, the type of computer you are connecting from.<br />
<br />
Note that your password should never, never, never be to sent to us, even if your question is about your account.<br />
<br />
Try to avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team!<br />
<br />
===What does ''code scaling'' mean?===<br />
<br />
'''Answer:'''<br />
<br />
Please see [[Introduction_To_Performance#Parallel_Speedup|A Performance Primer]]<br />
<br />
===What do you mean by ''throughput''?===<br />
<br />
'''Answer:'''<br />
<br />
Please see [[Introduction_To_Performance#Throughput|A Performance Primer]].<br />
<br />
Here is a simple example:<br />
<br />
Suppose you need to do 10 computations. Say each of these runs for<br />
1 day on 8 cores, but they take "only" 18 hours on 16 cores. What is the<br />
fastest way to get all 10 computations done - as 8-core jobs or as<br />
16-core jobs? Let us assume you have 2 nodes at your disposal.<br />
The answer, after some simple arithmetic, is that running your 10<br />
jobs as 8-core jobs will take 5 days, whereas if you ran them<br />
as 16-core jobs it would take 7.5 days. Take your own conclusions...<br />
<br />
===I changed my .bashrc/.bash_profile and now nothing works===<br />
<br />
The default startup scripts provided by SciNet, and guidelines for them, can be found [[Important_.bashrc_guidelines|here]]. Certain things - like sourcing <tt>/etc/profile</tt><br />
and <tt>/etc/bashrc</tt> are ''required'' for various SciNet routines to work! <br />
<br />
If the situation is so bad that you cannot even log in, please send email [mailto:support@scinet.utoronto.ca support].<br />
<br />
===Could I have my login shell changed to (t)csh?===<br />
<br />
The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" <tt>#!/bin/tcsh</tt> at the top.<br />
<br />
===How can I run Matlab / IDL / Gaussian / my favourite commercial software at SciNet?===<br />
<br />
'''Answer:'''<br />
<br />
Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).<br />
<br />
If your research group requires a commercial package that you already have or are willing to buy licenses for, contact us at [mailto:support@scinet.utoronto.ca support@scinet] and we can work together to find out if it is feasible to implement the packages licensing arrangement on the SciNet clusters, and if so, what is the the best way to do it.<br />
<br />
Note that it is important that you contact us before installing commercially licensed software on SciNet machines, even if you have a way to do it in your own directory without requiring sysadmin intervention. It puts us in a very awkward position if someone is found to be running unlicensed or invalidly licensed software on our systems, so we need to be aware of what is being installed where.<br />
<br />
===Do you have a recommended ssh program that will allow scinet access from Windows machines?===<br />
<br />
'''Answer:'''<br />
<br />
The [[Ssh#SSH_for_Windows_Users | SSH for Windows users]] programs we recommend are:<br />
<br />
* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools, including ssh and X, all wrapped up into one executable.<br />
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] - this is a terminal for windows that connects via ssh. It is a quick install and will get you up and running quickly.<br/> '''WARNING:''' Make sure you download putty from the official website, because there are "trojanized" versions of putty around that will send your login information to a site in Russia (as reported [http://blogs.cisco.com/security/trojanized-putty-software here]).<br>To set up your passphrase protected ssh key with putty, see [http://the.earth.li/~sgtatham/putty/0.61/htmldoc/Chapter8.html#pubkey here].<br />
* [http://www.cygwin.com/ CygWin] - this is a whole linux-like environment for windows, which also includes an X window server so that you can display remote windows on your desktop. Make sure you include the openssh and X window system in the installation for full functionality. This is recommended if you will be doing a lot of work on Linux machines, as it makes a very similar environment available on your computer.<br>To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.<br />
<br>To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.<br />
<br />
===My ssh key does not work! WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ===<br />
<br />
'''Answer:'''<br />
<br />
[[Ssh_keys#Testing_Your_Key | Testing Your Key]]<br />
<br />
* If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.<br />
<pre><br />
$ ssh USERNAME@login.scinet.utoronto.ca<br />
<br />
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br />
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @<br />
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br />
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!<br />
Someone could be eavesdropping on you right now (man-in-the-middle<br />
attack)!<br />
It is also possible that the RSA host key has just been changed.<br />
The fingerprint for the RSA key sent by the remote host is<br />
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.<br />
Please contact your system administrator.<br />
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of<br />
this message.<br />
Offending key in /home/<user>/.ssh/known_hosts:3<br />
RSA host key for login.scinet.utoronto.ca <br />
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has<br />
changed and you have requested<br />
</pre><br />
<br />
* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be<br />
restarted with the new passphrase ssh key.<br />
<pre><br />
$ ssh USERNAME@login.scinet.utoronto.ca<br />
<br />
Agent admitted failure to sign using the key.<br />
</pre><br />
<br />
===Can't get graphics: "Can't open display/DISPLAY is not set"===<br />
<br />
To use graphics on SciNet machines and have it displayed on your machine, you need to have a X server running on your computer (an X server is the standard way graphics is done on linux). One an X server is running, you can log in with the "-Y" option to ssh ("-X" sometimes also works).<br />
<br />
How to get an X server running on your computer, depends on the operating system. On linux machines with a graphical interface, X will already be running. On windows, the easiest solution is using MobaXterm, which comes with an X server (alternatives, such as cygwin with the x11 server installed, or running putty+Xming, can also work, but are a bit more work to set up. For Macs, you will need to install Xquartz. <br />
<br />
===Remote graphics stops working after a while: "Can't open display"===<br />
<br />
If you still cannot get graphics, or it works only for a while and then suddenly it "can't open display localhost:....", your X11 graphics connection may have timed out (Macs seem to be particularly prone to this). You'll have to tell your own computer not to allow, and not to timeout the X11 graphics connection.<br />
<br />
The following should fix it. The ssh configuration settings are in a file called /etc/ssh/ssh_config (or /etc/ssh_config in older OS X versions, or $HOME/.ssh/config for specific users). In the config file, find (or create) the section "Host *" (meaning all hosts) and add the following lines:<br />
<br />
Host *<br />
ServerAliveInterval 60<br />
ServerAliveCountMax 3<br />
ForwardX11 yes<br />
ForwardX11Trusted yes<br />
ForwardX11Timeout 596h<br />
<br />
(The <tt>Host *</tt> is only needed if there was no Host section yet to append these settings to.)<br />
<br />
If this does not resolve it, try it again with "ssh -vvv -Y ....". The "-vvv" spews out a lot of diagnostic messages. Look for anything resembling a timeout, and let us know (support AT scinet DOT utoronto DOT ca).<br />
<br />
===Can't forward X: "Warning: No xauth data; using fake authentication data", or "X11 connection rejected because of wrong authentication."===<br />
<br />
I used to be able to forward X11 windows from SciNet to my home machine, but now I'm getting these messages; what's wrong?<br />
<br />
'''Answer:'''<br />
<br />
This very likely means that ssh/xauth can't update your ${HOME}/.Xauthority file. <br />
<br />
The simplest pssible reason for this is that you've filled your 10GB /home quota and so can't write anything to your home directory. Use<br />
<br />
<pre><br />
$ module load extras<br />
$ diskUsage<br />
</pre> <br />
<br />
to check to see how close you are to your disk usage on ${HOME}.<br />
<br />
Alternately, this could mean your .Xauthority file has become broken/corrupted/confused some how, in which case you can delete that file, and when you next log in you'll get a similar warning message involving creating .Xauthority, but things should work.<br />
<br />
===I have a CCDB account, but I can't login to SciNet. How can I get a SciNet account?===<br />
<br />
'''Answer:'''<br />
<br />
You must extend your CCDB application process to also get a SciNet account:<br />
<br />
https://wiki.scinet.utoronto.ca/wiki/index.php/Application_Process<br />
<br />
https://www.scinethpc.ca/getting-a-scinet-account/<br />
<br />
<br />
===How come I can not login to TCS?===<br />
<br />
'''Answer:'''<br />
<br />
A SciNet account doesn't automatically entitle you to TCS access. At a minimum, TCS jobs need to run on at least 32 cores (64 preferred because of Simultaneous Multi Threading - [[TCS_Quickstart#Node_configuration|SMT]] - on these nodes) and need the large memory (4GB/core) and bandwidth on the system. Essentially you need to be able to explain why the work can't be done on the GPC.<br />
<br />
===How can I reset the password for my Compute Canada account?===<br />
<br />
'''Answer:'''<br />
<br />
You can reset your password for your Compute Canada account here:<br />
<br />
https://ccdb.computecanada.ca/security/forgot<br />
<br />
<br />
===How can I change or reset the password for my SciNet account?===<br />
<br />
'''Answer:'''<br />
<br />
To reset your password at SciNet please go to [https://portal.scinet.utoronto.ca/password_resets Password reset page].<br />
<br />
If you know your old password and want to change it, that can be done here after logging in on the portal:<br />
<br />
https://portal.scinet.utoronto.ca<br />
<br />
===Why am I getting the error "Permission denied (publickey,gssapi-with-mic,password)"?===<br />
<br />
This error can pop up in a variety of situations: when trying to log in, or when after a job has finished, when the error and output files fail to be copied (there are other possible reasons for this failure as well -- see [[FAQ#My_GPC_job_died.2C_telling_me_.60Copy_Stageout_Files_Failed.27|My GPC job died, telling me:Copy Stageout Files Failed]]).<br />
In most cases, the "Permission denioed" error is caused by incorrect permission of the (hidden) .ssh directory. Ssh is used for logging in as well as for the copying of the standard error and output files after a job. <br />
<br />
For security reasons, <br />
the directory .ssh should only be writable and readable to you, but yours <br />
has read permission for everybody, and thus it fails. You can change <br />
this by<br />
<pre><br />
chmod 700 ~/.ssh<br />
</pre><br />
And to be sure, also do<br />
<pre><br />
chmod 600 ~/.ssh/id_rsa ~/authorized_keys<br />
</pre><br />
<br />
===ERROR:102: Tcl command execution failed? when loading modules ===<br />
Modules sometimes require other modules to be loaded first.<br />
Module will let you know if you didn’t.<br />
For example:<br />
<pre><br />
$ module purge<br />
$ module load python<br />
python/2.6.2(11):ERROR:151: Module ’python/2.6.2’ depends on one of the module(s) ’gcc/4.4.0’<br />
python/2.6.2(11):ERROR:102: Tcl command execution failed: prereq gcc/4.4.0<br />
$ gpc-f103n084-$ module load gcc python<br />
$<br />
</pre><br />
<br />
<br />
<br />
=== How do I compute the core-years usage of my code? ===<br />
<br />
The "core-years" quantity is a way to account for the time your code runs, by considering the total number of cores and time used, accounting for the total number of hours in a year.<br />
For instance if your code uses ''HH'' hours, in ''NN'' nodes, where each node has ''CC'' cores, then "core-years" can be computed as follow:<br />
<br />
''HH*(NN*CC)/(365*24)''<br />
<br />
If you have several independent instances (batches) running on different nodes, with ''BB'' number of batches and each batch during ''HH'' hours, then your core-years usage can be computed as,<br />
<br />
''BB*HH*(NN*CC)/(365*24)''<br />
<br />
As a general rule, in our GPC system, each node has only 8 cores, so ''CC'' will be always 8.<br />
<br />
==Compiling your Code==<br />
<br />
===How can I get g77 to work?===<br />
<br />
The fortran 77 compilers on the GPC are ifort and gfortran. We have dropped support for g77. This has been a conscious decision. g77 (and the associated library libg2c) were completely replaced six years ago (Apr 2005) by the gcc 4.x branch, and haven't undergone any updates at all, even bug fixes, for over five years. <br />
If we would install g77 and libg2c, we would have to deal with the inevitable confusion caused when users accidentally link against the old, broken, wrong versions of the gcc libraries instead of the correct current versions. <br />
<br />
If your code for some reason specifically requires five-plus-year-old libraries, availability, compatibility, and unfixed-known-bug problems are only going to get worse for you over time, and this might be as good an opportunity as any to address those issues. <br />
<br />
''A note on porting to gfortran or ifort:''<br />
<br />
While gfortran and ifort are rather compatible with g77, one <br />
important difference is that by default, gfortran does not preserve <br />
local variables between function calls, while g77 does. Preserved <br />
local variables are for instance often used in implementations of quasi-random number <br />
generators. Proper fortran requires to declare such variables as SAVE <br />
but not all old code does this.<br />
Luckily, you can change gfortran's default behavior with the flag <br />
<tt>-fno-automatic</tt>. For ifort, the corresponding flag is <tt>-noautomatic</tt>.<br />
<br />
===Where is libg2c.so?===<br />
<br />
libg2c.so is part of the g77 compiler, for which we dropped support. See [[#How can I get g77 to work on the GPC?]] for our reasons.<br />
<br />
===Autoparallelization does not work!===<br />
<br />
I compiled my code with the <tt>-qsmp=omp,auto</tt> option, and then I specified that it should be run with 64 threads - with <br />
export OMP_NUM_THREADS=64<br />
<br />
However, when I check the load using <tt>llq1 -n</tt>, it shows a load on the node of 1.37. Why?<br />
<br />
'''Answer:'''<br />
<br />
Using the autoparallelization will only get you so far. In fact, it usually does not do too much. What is helpful is to run the compiler with the <tt>-qreport</tt> option, and then read the output listing carefully to see where the compiler thought it could parallelize, where it could not, and the reasons for this. Then you can go back to your code and carefully try to address each of the issues brought up by the compiler.<br />
We ''emphasize'' that this is just a rough first guide, and that the compilers are still not magical! For more sophisticated approaches to parallelizing your code, email us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] to set up an appointment with one<br />
of our technical analysts.<br />
<br />
===How do I link against the Intel Math Kernel Library?===<br />
<br />
If you need to link to the Intel Math Kernal Library (MKL) with the intel compilers, just add the <pre>-mkl</pre> flag. There are in fact three flavours: <tt>-mkl=sequential</tt>, <tt>-mkl=parallel</tt> and <tt>-mkl=cluster</tt>, for the serial version, the threaded version and the mpi version, respectively. (Note: The cluster version is available only when using the intelmpi module and mpi compilation wrappers.)<br />
<br />
If you need to link in the Intel Math Kernel Library (MKL) libraries to gcc/gfortran/c++, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code.<br />
<br />
'''''Note that this give the link line for the command line. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''''<br />
<br />
'''''Note too that, unless the integer arguments you will be passing to the MKL libraries are actually 64-bit integers, rather than the normal int or INTEGER types, you want to specify 32-bit integers (lp64) .'''''<br />
<br />
===Can the compilers on the login nodes be disabled to prevent accidentally using them?===<br />
<br />
'''Answer:'''<br />
<br />
You can accomplish this by modifying your .bashrc to not load the compiler modules. See [[Important .bashrc guidelines]].<br />
<br />
==="relocation truncated to fit: R_X86_64_PC32": Huh?===<br />
<br />
What does this mean, and why can't I compile this code?<br />
<br />
'''Answer:'''<br />
<br />
Welcome to the joys of the x86 architecture! You're probably having trouble building arrays larger than 2GB, individually or together. Generally, you have to try to use the medium or large x86 `memory model'. For the intel compilers, this is specified with the compile options<br />
<br />
-mcmodel=medium -shared-intel<br />
<br />
==="feupdateenv is not implemented and will always fail"===<br />
<br />
How do I get rid of this and what does it mean?<br />
<br />
'''Answer:'''<br />
First note that, as ominous as it sounds, this is really just a warning, and has to do with the intel math library. You can ignore it (unless you really are trying to manually change the exception handlers for floating point exceptions such as divide by zero), or take the safe road and get rid off it by linking with the intel math functions library:<pre>-limf</pre>See also [[#How do I link against the Intel Math Kernel Library?]]<br />
<br />
===Cannot find rdmacm library when compiling on GPC===<br />
<br />
I get the following error building my code on GPC: "<tt>ld: cannot find -lrdmacm</tt>". Where can I find this library?<br />
<br />
'''Answer:'''<br />
<br />
This library is part of the MPI libraries; if your compiler is having problems picking it up, it probably means you are mistakenly trying to compile on the login nodes (scinet01..scinet04). The login nodes aren't part of the GPC; they are for logging into the data centre only. From there you must go to the GPC or TCS development nodes to do any real work.<br />
<br />
=== Why do I get this error when I try to compile: "icpc: error #10001: could not find directory in which /usr/bin/g++41 resides" ?===<br />
<br />
You are trying to compile on the login nodes. As described in the wiki ( https://support.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Login ), or in the users guide you would have received with your account, Scinet supports two main clusters, with very different architectures. Compilation must be done on the development nodes of the appropriate cluster (in this case, gpc01-04). Thus, log into gpc01, gpc02, gpc03, or gpc04, and compile from there.<br />
<br />
==Testing your Code==<br />
<br />
=== Can I run a something for a short time on the development nodes? ===<br />
<br />
I am in the process of playing around with the mpi calls in my code to get it to work. I do a lot of tests and each of them takes a couple of seconds only. Can I do this on the development nodes?<br />
<br />
'''Answer:'''<br />
<br />
Yes, as long as it's very brief (a few minutes). People use the development nodes<br />
for their work, and you don't want to bog it down for people, and testing a real<br />
code can chew up a lot more resources than compiling, etc. The procedures differ<br />
depending on what machine you're using.<br />
<br />
==== TCS ====<br />
<br />
On the TCS you can run small MPI jobs on the tcs02 node, which is meant for <br />
development use. But even for this test run on one node, you'll need a host file --<br />
a list of hosts (in this case, all tcs-f11n06, which is the `real' name of tcs02)<br />
that the job will run on. Create a file called `hostfile' containing the following:<br />
<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
tcs-f11n06<br />
<br />
for a 4-task run. When you invoke "poe" or "mpirun", there are runtime<br />
arguments that you specify pointing to this file. You can also specify it<br />
in an environment variable MP_HOSTFILE, so, if your file is in your /scratch directory, say <br />
${SCRATCH}/hostfile, then you would do<br />
<br />
<pre><br />
export MP_HOSTFILE=${SCRATCH}/hostfile<br />
</pre><br />
<br />
in your shell. You will also need to create a <tt>.rhosts</tt> file in your <br />
home director, again listing <tt>tcs-f11n06</tt> so that <tt>poe</tt><br />
can start jobs. After that you can simply run your program. You can use<br />
mpiexec:<br />
<br />
<pre><br />
mpiexec -n 4 my_test_program<br />
</pre><br />
<br />
adding <tt> -hostfile /path/to/my/hostfile</tt> if you did not set the environment<br />
variable above. Alternatively, you can run it with the poe command (do a "man poe" for details), or even by<br />
just directly running it. In this case the number of MPI processes will by default<br />
be the number of entries in your hostfile.<br />
<br />
<br />
==== GPC ====<br />
<br />
On the GPC one can run short test jobs on the GPC [[GPC_Quickstart#Compile.2FDevel_Nodes | development nodes ]]<tt>gpc01</tt>..<tt>gpc04</tt>;<br />
if they are single-node jobs (which they should be) they don't need a hostfile. Even better, though, is to request an [[ Moab#Interactive | interactive ]] job and run the tests either in regular batch queue or using a short high availability [[ Moab#debug | debug ]] queue that is reserved for this purpose.<br />
<br />
=== How do I run a longer (but still shorter than an hour) test job quickly ? ===<br />
<br />
'''Answer'''<br />
<br />
On the GPC there is a high turnover short queue called [[ Moab#debug | debug ]] that is designed for<br />
this purpose. You can use it by adding <br />
<pre><br />
#PBS -q debug<br />
</pre><br />
to your submission script.<br />
<br />
==Submitting your jobs==<br />
<br />
===Error Submitting My Job: qsub: Bad UID for job execution MSG=ruserok failed ===<br />
<br />
I write up a submission script as in the examples, but when I attempt to submit the job, I get the above error. What's wrong?<br />
<br />
'''Answer:'''<br />
<br />
This error will occur if you try to submit a job from the login nodes. The login nodes are the gateway to all of SciNet's systems (GPC, TCS, P7, ARC), which have different hardware and queueing systems. To submit a job, you must log into a development node for the particular cluster you are submitting to and submit from there.<br />
<br />
<br />
=== How do I charge jobs to my RAC allocation? ===<br />
<br />
'''Answer:'''<br />
<br />
Please see the [[Moab#Accounting|accounting section of Moab page]].<br />
<br />
<br />
===How can I automatically resubmit a job?===<br />
<br />
Commonly you may have a job that you know will take longer to run than what is <br />
permissible in the queue. As long as your program contains [[Checkpoints|checkpoint]] or <br />
restart capability, you can have one job automatically submit the next. In<br />
the following example it is assumed that the program finishes before <br />
the 48 hour limit and then resubmits itself by logging into one<br />
of the development nodes.<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque example submission script for auto resubmission<br />
# SciNet GPC<br />
#<br />
#PBS -l nodes=1:ppn=8,walltime=48:00:00<br />
#PBS -N my_job<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
# YOUR CODE HERE<br />
./run_my_code<br />
<br />
# RESUBMIT 10 TIMES HERE<br />
num=$NUM<br />
if [ "$num" -lt 10 ]; then<br />
num=$(($num+1))<br />
ssh gpc01 "cd $PBS_O_WORKDIR; qsub ./script_name.sh -v NUM=$num";<br />
fi<br />
</source><br />
<br />
<pre><br />
qsub script_name.sh -v NUM=0<br />
</pre><br />
<br />
You can alternatively use [[ Moab#Job_Dependencies | Job dependencies ]] through the queuing system which will not start one job until another job has completed.<br />
<br />
If your job can't be made to automatically stop before the 48 hour queue window, but it does write out checkpoints, you can use the timeout command to stop the program while you still have time to resubmit; for instance<br />
<br />
<source lang="bash"><br />
timeout 2850m ./run_my_code argument1 argument2<br />
</source><br />
<br />
will run the program for 47.5 hours (2850 minutes), and then send it SIGTERM to exit the program.<br />
<br />
===How can I pass in arguments to my submission script?===<br />
<br />
If you wish to make your scripts more generic you can use qsub's ability <br />
to pass in environment variables to pass in arguments to your script.<br />
The following example shows a case where an input and an output <br />
file are passed in on the qsub line. Multiple variables can be <br />
passed in using the qsub "-v" option and comma delimited. <br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
# MOAB/Torque example of passing in arguments<br />
# SciNet GPC<br />
# <br />
#PBS -l nodes=1:ppn=8,walltime=48:00:00<br />
#PBS -N my_job<br />
<br />
# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from<br />
cd $PBS_O_WORKDIR<br />
<br />
# YOUR CODE HERE<br />
./run_my_code -f $INFILE -o $OUTFILE<br />
</source><br />
<br />
<pre><br />
qsub script_name.sh -v INFILE=input.txt,OUTFILE=outfile.txt<br />
</pre><br />
<br />
<br />
===I submit my GPC job, and I get an email saying it was rejected===<br />
'''Answer:'''<br />
<br />
This happens because the job you've submitted breaks one of the rules of the queues and is rejected. An email<br />
is sent with the JOBID, JOBNAME, and the reason it was rejected. The following is an example where a job<br />
requests more than 48 hours and was rejected.<br />
<br />
<pre><br />
PBS Job Id: 3462493.gpc-sched<br />
Job Name: STDIN<br />
job deleted<br />
Job deleted at request of root@gpc-sched<br />
MOAB_INFO: job was rejected - job violates class configuration 'wclimit too high for class 'batch_ib' (345600 > 172800)'<br />
</pre><br />
<br />
Jobs on the TCS or GPC may only run for 48 hours at a time; this restriction greatly increases responsiveness of the queue and queue throughput for all our users. If your computation requires longer than that, as many do, you will have to [[ Checkpoints | checkpoint ]] your job and restart it after each 48-hour queue window. You can manually re-submit jobs, or if you can have your job cleanly exit before the 48 hour window, there are ways to [[ FAQ#How_can_I_automatically_resubmit_a_job.3F | automatically resubmit jobs ]].<br />
<br />
Other rejections return a more cryptic error saying "job violates class configuration" such as follows:<br />
<pre><br />
PBS Job Id: 3462409.gpc-sched<br />
Job Name: STDIN<br />
job deleted<br />
Job deleted at request of root@gpc-sched<br />
MOAB_INFO: job was rejected - job violates class configuration 'user required by class 'batch''<br />
</pre><br />
<br />
The most common problems that result in this error are:<br />
<br />
* '''Incorrect number of processors per node''': Jobs on the GPC are scheduled per-node not per-core and since each node has 8 processor cores (ppn=8) the smallest job allowed is one node with 8 cores (nodes=1:ppn=8). For serial jobs users must bundle or batch them together in groups of 8. See [[ FAQ#How_do_I_run_serial_jobs_on_GPC.3F | How do I run serial jobs on GPC? ]]<br />
* '''No number of nodes specified''': Jobs submitted to the main queue must request a specific number of nodes, either in the submission script (with a line like <tt>#PBS -l nodes=2:ppn=8</tt>) or on the command line (eg, <tt>qsub -l nodes=2:ppn=8,walltime=5:00:00 script.pbs</tt>). Note that for the debug queue, you can get away without specifying a number of nodes and a default of one will be assigned; for both technical and policy reasons, we do not enforce such a default for the main ("batch") queue.<br />
* '''There is a 15 minute walltime minimum''' on all queues except debug and if you set your walltime less than this, it will be rejected.<br />
<br />
<br />
<br />
=== When submitting your job, fails saying: "script is written in DOS/Windows text format" ===<br />
<br />
'''Answer:'''<br />
<br />
Very likely you have written your script in a windows machine, so to fix this you just need to change the format of you submission script to unix from Windows/DOS.<br />
Use the command below for all your script files:<br />
<br />
<pre><br />
dos2unix <pbs-script-file><br />
</pre><br />
<br />
where <pbs-script-file> has to be substituted by the name of your script file.<br />
<br />
==Running your jobs==<br />
<br />
===My job can't write to /home===<br />
<br />
My code works fine when I test on the development nodes, but when I submit a job, or even run interactively in the development queue on GPC, it fails. What's wrong?<br />
<br />
'''Answer:'''<br />
<br />
As [[Data_Management#Home_Disk_Space | discussed]] [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf elsewhere], <tt>/home</tt> is mounted read-only on the compute nodes; you can only write to <tt>/home</tt> from the login nodes and devel nodes. (The [[GPC_Quickstart#128Glargemem | largemem nodes]] on GPC, in this respect, are more like devel nodes than compute nodes). In general, to run jobs you can read from <tt>/home</tt> but you'll have to write to <tt>/scratch</tt> (or, if you were allocated space through the RAC process, on <tt>/project</tt>). More information on SciNet filesytems can be found on our [[Data_Management | Data Management]] page.<br />
<br />
===OpenMP on the TCS===<br />
<br />
How do I run an OpenMP job on the TCS?<br />
<br />
'''Answer:'''<br />
<br />
Please look at the [[TCS_Quickstart#Submission_Script_for_an_OpenMP_Job | TCS Quickstart ]] page.<br />
<br />
===Can I can use hybrid codes consisting of MPI and openMP on the GPC?===<br />
<br />
'''Answer:'''<br />
<br />
Yes. Please look at the [[GPC_Quickstart#Hybrid_MPI.2FOpenMP_jobs | GPC Quickstart ]] page.<br />
<br />
===How do I run serial jobs on GPC?===<br />
<br />
'''Answer''':<br />
<br />
So it should be said first that SciNet is a parallel computing resource, <br />
and our priority will always be parallel jobs. Having said that, if <br />
you can make efficient use of the resources using serial jobs and get <br />
good science done, that's good too, and we're happy to help you.<br />
<br />
The GPC nodes each have 8 processing cores, and making efficient use of these <br />
nodes means using all eight cores. As a result, we'd like to have the <br />
users take up whole nodes (eg, run multiples of 8 jobs) at a time. <br />
<br />
It depends on the nature of your job what the best strategy is. Several approaches are presented on the [[User_Serial|serial run wiki page]].<br />
<br />
===Why can't I request only a single cpu for my job on GPC?===<br />
<br />
'''Answer''':<br />
<br />
On GPC, computers are allocated by the node - that is, in chunks of 8 processors. If you want to run a job that requires only one processor, you need to bundle the jobs into groups of 8, so as to not be wasting the other 7 for 48 hours. See [[User_Serial|serial run wiki page]].<br />
<br />
===How do I run serial jobs on TCS?===<br />
<br />
'''Answer''': You don't.<br />
<br />
<br />
===But in the queue I found a user who is running jobs on GPC, each of which is using only one processor, so why can't I?===<br />
<br />
'''Answer''':<br />
<br />
The pradat* and atlaspt* jobs, amongst others, are jobs of the ATLAS high energy physics project. That they are reported as single cpu jobs is an artifact of the moab scheduler. They are in fact being automatically bundled into 8-job bundles but have to run individually to be compatible with their international grid-based systems.<br />
<br />
===How do I use the ramdisk on GPC?===<br />
<br />
To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) ${SCRATCH}. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.<br />
<br />
It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.<br />
<br />
''More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].''<br />
<br />
<br />
=== How can I run a job longer than 48 hours? ===<br />
<br />
'''Answer:'''<br />
<br />
The SciNet queues have a queue limit of 48 hours. This is pretty typical for systems of its size in Canada and elsewhere, and larger systems commonly have shorter limits. The limits are there to ensure that every user gets a fair share of the system (so that no one user ties up lots of nodes for a long time), and for safety (so that if one memory board in one node fails in the middle of a very long job, you haven't lost a months' worth of work).<br />
<br />
Since many of us have simulations that require more than that much time, most widely-used scientific applications have "checkpoint-restart" functionality, where every so often the complete state of the calculation is stored as a checkpoint file, and one can restart a simulation from one of these. In fact, these restart files tend to be quite useful for a number of purposes.<br />
<br />
If your job will take longer, you will have to submit your job in multiple parts, restarting from a checkpoint each time. In this way, one can run a simulation much longer than the queue limit. In fact, one can even write job scripts which automatically re-submit themselves until a run is completed, using [[FAQ#How_can_I_automatically_resubmit_a_job.3F | automatic resubmission. ]]<br />
<br />
=== Why did showstart say it would take 3 hours for my job to start before, and now it says my job will start in 10 hours? ===<br />
<br />
'''Answer:'''<br />
<br />
Please look at the [[FAQ#How_do_priorities_work.2Fwhy_did_that_job_jump_ahead_of_mine_in_the_queue.3F | How do priorities work/why did that job jump ahead of mine in the queue? ]] page.<br />
<br />
===How do priorities work/why did that job jump ahead of mine in the queue?===<br />
<br />
'''Answer:'''<br />
<br />
The [[Moab | queueing system]] used on SciNet machines is a [http://en.wikipedia.org/wiki/Priority_queue Priority Queue]. Jobs enter the queue at the back of the queue, and slowly make their way to the front as those ahead of them are run; but a job that enters the queue with a higher priority can `cut in line'.<br />
<br />
The main factor which determines priority is whether or not the user (or their PI) has an [http://wiki.scinethpc.ca/wiki/index.php/Application_Process RAC allocation]. These are competitively allocated grants of computer time; there is a call for proposals towards the end of every calendar year. Users with an allocation have high priorities in an attempt to make sure that they can use the amount of computer time the committees granted them. Their priority decreases as they approach their allotted usage over the current window of time; by the time that they have exhausted that allotted usage, their priority is the same as users with no allocation (unallocated, or `default' users). Unallocated users have a fixed, low, priority.<br />
<br />
This priority system is called `fairshare'; the scheduler attempts to make sure everyone has their fair share of the machines, where the share that's fair has been determined by the allocation committee. The fairshare window is a rolling window of two weeks; that is, any time you have a job in the queue, the fairshare calculation of its priority is given by how much of your allocation of the machine has been used in the last 14 days.<br />
<br />
A particular allocation might have some fraction of GPC - say 4% of the machine (if the PI had been allocated 10 million CPU hours on GPC). The allocations have labels; (called `Resource Allocation Proposal Identifiers', or RAPIs) they look something like<br />
<br />
abc-123-ab<br />
<br />
where abc-123 is the PIs CCRI, and the suffix specifies which of the allocations granted to the PI is to be used. These can be specified on a job-by-job basis. On GPC, one adds the line<br />
#PBS -A RAPI<br />
to your script; on TCS, one uses<br />
# @ account_no = RAPI<br />
If the allocation to charge isn't specified, a default is used; each user has such a default, which can be changed at the same portal where one changes one's password, <br />
<br />
https://portal.scinet.utoronto.ca/<br />
<br />
A jobs priority is determined primarily by the fairshare priority of the allocation it is being charged to; the previous 14 days worth of use under that allocation is calculated and compared to the allocated fraction (here, 5%) of the machine over that window (here, 14 days). The fairshare priority is a decreasing function of the allocation left; if there is no allocation left (eg, jobs running under that allocation have already used 379,038 CPU hours in the past 14 days), the priority is the same as that of a user with no granted allocation. (This last part has been the topic of some debate; as the machine gets more utilized, it will probably be the case that we allow RAC users who have greatly overused their quota to have their priorities to drop below that of unallocated users, to give the unallocated users some chance to run on our increasingly crowded system; this would have no undue effect on our allocated users as they still would be able to use the amount of resources they had been allocated by the committees.) Note that all jobs charging the same allocation get the same fairshare priority.<br />
<br />
There are other factors that go into calculating priority, but fairshare is the most significant. Other factors include<br />
* amount of time waiting in queue (measured in units of the requested runtime). A waiting queue job gains priority as it sits in the queue to avoid job starvation. <br />
* User adjustment of priorities ( See below ).<br />
<br />
The major effect of these subdominant terms is to shuffle the order of jobs running under the same allocation.<br />
<br />
===How do we manage job priorities within our research group?===<br />
<br />
'''Answer:'''<br />
<br />
Obviously, managing shared resources within a large group - whether it <br />
is conference funding or CPU time - takes some doing. <br />
<br />
It's important to note that the fairshare periods are intentionally kept <br />
quite short - just two weeks long. So, for example, let us say that in your resource <br />
allocation you have about 10% of the machine. Then for someone to use <br />
up the whole two week amount of time in 2 days, they'd have to use 70% <br />
of the machine in those two days - which is unlikely to happen by <br />
accident. If that does happen, <br />
those using the same allocation as the person who used 70% of the <br />
machine over the two days will suffer by having much lower priority for <br />
their jobs, but only for the next 12 days - and even then, if there are <br />
idle cpus they'll still be able to compute.<br />
<br />
There will be online tools for seeing how the allocation is being used, <br />
and those people who are in charge in your group will be able to use <br />
that information to manage the users, telling them to dial it down or <br />
up. We know that managing a large research group is hard, and we want <br />
to make sure we provide you the information you need to do your job <br />
effectively.<br />
<br />
One way for users within a group to manage their priorities within the group<br />
is with [[Moab#Adjusting_Job_Priority | user-adjusted priorities]]; this is<br />
described in more detail on the [[Moab | Scheduling System]] page.<br />
<br />
<br />
<br />
<!--<br />
==Errors in running jobs==<br />
--><br />
<br />
=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===<br />
<br />
On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the<br />
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,<br />
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in<br />
/var/spool/torque/spool.<br />
<br />
This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.<br />
<br />
Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.<br />
<br />
=== My GPC job died, telling me `Copy Stageout Files Failed' ===<br />
<br />
'''Answer:'''<br />
<br />
When a job runs on GPC, the script's standard output and error are redirected to <br />
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in<br />
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)<br />
<br />
When you get errors like this:<br />
<pre><br />
An error has occurred processing your job, see below.<br />
request to copy stageout files failed on node<br />
</pre><br />
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.<br />
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process (e.g. ran out of memory very quickly) and those files just never got copied.<br />
<br />
Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant). <br />
<br />
Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.<br />
<br />
<!--<br />
<br />
===Another transport will be used instead===<br />
<br />
I get error messages like the following when running on the GPC at the start of the run, although the job seems to proceed OK. Is this a problem?<br />
<pre><br />
--------------------------------------------------------------------------<br />
[[45588,1],0]: A high-performance Open MPI point-to-point messaging module<br />
was unable to find any relevant network interfaces:<br />
<br />
Module: OpenFabrics (openib)<br />
Host: gpc-f101n005<br />
<br />
Another transport will be used instead, although this may result in<br />
lower performance.<br />
--------------------------------------------------------------------------<br />
</pre><br />
<br />
'''Answer:'''<br />
<br />
Everything's fine. The two MPI libraries scinet provides work for both the InifiniBand and the Gigabit Ethernet interconnects, and will always try to use the fastest interconnect available. In this case, you ran on normal gigabit GPC nodes with no infiniband; but the MPI libraries have no way of knowing this, and try the infiniband first anyway. This is just a harmless `failover' message; it tried to use the infiniband, which doesn't exist on this node, then fell back on using Gigabit ethernet (`another transport').<br />
<br />
With OpenMPI, this can be avoided by not looking for infiniband; eg, by using the option<br />
<br />
--mca btl ^openib<br />
<br />
--><br />
<br />
===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===<br />
<br />
Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.<br />
<br />
In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or <br />
network configurations may exhibit this problem at higher or lower MPI <br />
task counts.<br />
<br />
Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:<br />
<br />
<pre><br />
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288<br />
</pre><br />
<br />
With Intel MPI, you should be able to do<br />
<br />
<pre><br />
module load intelmpi/4.0.3.008<br />
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode<br />
</pre><br />
<br />
to the same end. <br />
<br />
For more information see [[GPC MPI Versions]].<br />
<br />
<br />
===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===<br />
<br />
'''Answer:'''<br />
<br />
To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.<br />
<br />
Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.<br />
<br />
==Monitoring jobs in the queue==<br />
<br />
<br />
===Why hasn't my job started?===<br />
<br />
'''Answer:'''<br />
<br />
Use the moab command <br />
<br />
<pre><br />
checkjob -v jobid<br />
</pre><br />
<br />
and the last couple of lines should explain why a job hasn't started. <br />
<br />
Please see [[Moab| Job Scheduling System (Moab) ]] for more detailed information<br />
<br />
===How do I figure out when my job will run?===<br />
<br />
'''Answer:'''<br />
<br />
Please see [[Moab#Available_Resources| Job Scheduling System (Moab) ]]<br />
<br />
<br />
<!-- ===My GPC job is Held, and checkjob says "Batch:PolicyViolation" ===<br />
<br />
'''Answer:'''<br />
<br />
When this happens, you'll see your job stuck in a BatchHold state. <br />
This happens because the job you've submitted breaks one of the rules of the queues, and is being held until you modify it or kill it and re-submit a conforming job. The most common problems are:<br />
--><br />
<br />
<br />
===Running checkjob on my job gives me messages about JobFail and rejected===<br />
<br />
Running checkjob on my job gives me messages that suggest my job has failed, as below: what did I do wrong?<br />
<br />
<pre><br />
AName: test<br />
State: Idle <br />
Creds: user:xxxxxx group:xxxxxxxx account:xxxxxxxx class:batch_ib qos:ibqos<br />
WallTime: 00:00:00 of 8:00:00<br />
BecameEligible: Wed Jul 23 10:39:27<br />
SubmitTime: Wed Jul 23 10:38:22<br />
(Time Queued Total: 00:01:47 Eligible: 00:01:05)<br />
<br />
Total Requested Tasks: 8<br />
<br />
Req[0] TaskCount: 8 Partition: ALL <br />
Opsys: centos6computeA Arch: --- Features: ---<br />
<br />
<br />
Notification Events: JobFail<br />
<br />
IWD: /scratch/x/xxxxxxxx/xxxxxxx/xxxxxxx<br />
Partition List: torque,DDR<br />
Flags: RESTARTABLE<br />
Attr: checkpoint<br />
StartPriority: 76<br />
rejected for Opsys - (null)<br />
rejected for State - (null)<br />
rejected for Reserved - (null)<br />
NOTE: job req cannot run in partition torque (available procs do not meet requirements : 0 of 8 procs found)<br />
idle procs: 793 feasible procs: 0<br />
<br />
Node Rejection Summary: [Opsys: 117][State: 2895][Reserved: 19]<br />
<br />
NOTE: job violates constraints for partition SANDY (partition SANDY not in job partition mask)<br />
<br />
NOTE: job violates constraints for partition GRAVITY (partition GRAVITY not in job partition mask)<br />
<br />
rejected for State - (null)<br />
NOTE: <br />
</pre><br />
<br />
'''Answer:'''<br />
<br />
The output from check job is a little cryptic in places, and if you are wondering why your job hasn't started yet, you might think that "rejection" and "JobFail" suggest that there's something wrong. But the above message is actually normal; you can use the <tt>showstart</tt> command on your job to get a (preliminary, subject to change) estimate as to when the job will start, and you'll find that it is in fact scheduled to start up in the near future.<br />
<br />
In the above message:<br />
<br />
* `Notification Events: JobFail` just means that, if notifications are enabled, you'll get a message if the job fails;<br />
* `job req cannot run in partition torque` just means that the job cannot run just yet (that's why it's queued);<br />
* `job req cannot run in dynamic partition DDR now (insufficient procs available: 0 < 8)` says why: there aren't processors available; and<br />
* `job violates constraints for partition SANDY/GRAVITY` just means that the job isn't eligable to run in those paritcular (small) sections of the cluster.<br />
<br />
that is, the above output is the normal and expected (if somewhat cryptic) explanation as to why the job is waiting - nothing to worry about.<br />
<br />
<br />
===How can I monitor my running jobs on TCS?===<br />
<br />
How can I monitor the load of TCS jobs?<br />
<br />
'''Answer:'''<br />
<br />
You can get more information with the command <br />
/xcat/tools/tcs-scripts/LL/jobState.sh<br />
which I alias as:<br />
alias llq1='/xcat/tools/tcs-scripts/LL/jobState.sh'<br />
If you run "llq1 -n" you will see a listing of jobs together with a lot of information, including the load.<br />
<br />
<br />
===How can I check the memory usage from my jobs?===<br />
<br />
How can I check the memory usage from my jobs?<br />
<br />
'''Answer:'''<br />
<br />
In many occasions it can be really useful to take a look at how much memory your job is using while it is running.<br />
There a couple of ways to do so:<br />
<br />
1) using some of the [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities command line utilities] we have developed, e.g: by using the '''jobperf''' or '''jobtop''' utilities, it will allow you to check the job performance and head's node utilization respectively.<br />
<br />
2) ''ssh'' into the nodes where your job is being run and check for memory usage and system stats right there. For instance, trying the 'top' or 'free' commands, in those nodes.<br />
<br />
Also, it always a good a idea and strongly encouraged to inspect the standard output-log and error-log generated for your job submissions.<br />
These files are named respectively: ''JobName.{o|e}jobIdNumber''; where ''JobName'' is the name you gave to the job (via the '-N' PBS flag) and ''JobIdNumber'' is the id number of the job.<br />
These files are saved in the working directory after the job is finished, but they can be also accessed on real-time using the '''jobError''' and '''jobOutput''' [https://wiki.scinet.utoronto.ca/wiki/index.php/SciNet_Command_Line_Utilities command line utilities] available loading the ''extras'' module.<br />
<br />
Other related topics to memory usage: <br><br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Ram_Disk Using Ram Disk]<br />
<br><br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Memory_Configuration Different Memory Configuration nodes]<br />
<br><br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/FAQ#Monitoring_jobs_in_the_queue Monitoring Jobs in the Queue]<br />
<br><br />
[https://wiki.scinet.utoronto.ca/wiki/images/a/a0/TechTalkJobMonitoring.pdf Tech Talk on Monitoring Jobs]<br />
<br />
<br />
===Can I run cron jobs on devel nodes to monitor my jobs?===<br />
<br />
Can I run cron jobs on devel nodes to monitor my jobs?<br />
<br />
'''Answer:'''<br />
<br />
No, we do not permit cron jobs to be run by users. To monitor the status of your jobs using a cron job running on your own machine, use the command<br />
<br />
<pre><br />
ssh myusername@login.scinet.utoronto.ca "qstat -u myusername"<br />
</pre><br />
<br />
or some variation of this command. Of course, you will need to have SSH keys setup on the machine running the cron job, so that password entry won't be necessary.<br />
<br />
<br />
=== How does one check the amount of used CPU-hours in a project, and how does one get statistics for each user in the project? ===<br />
<br />
'''Answer:'''<br />
<br />
This information is available on the scinet portal,https://portal.scinet.utoronto.ca, See also [[SciNet Usage Reports]].<br />
<br />
<!--<br />
==Errors in running jobs==<br />
<br />
<!--<br />
===On GPC, `Job cannot be executed'===<br />
<br />
I get error messages like this trying to run on GPC:<br />
<br />
<pre><br />
PBS Job Id: 30414.gpc-sched<br />
Job Name: namd<br />
Exec host: gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0<br />
Aborted by PBS Server <br />
Job cannot be executed<br />
See Administrator for help<br />
<br />
<br />
<br />
PBS Job Id: 30414.gpc-sched<br />
Job Name: namd<br />
Exec host: gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0<br />
An error has occurred processing your job, see below.<br />
request to copy stageout files failed on node 'gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0' for job 30414.gpc-sched<br />
<br />
Unable to copy file 30414.gpc-sched.OU to USER@gpc-f101n084.scinet.local:/scratch/G/GROUP/USER/projects/sim-performance-test/runtime/l/namd/8/namd.o30414<br />
*** error from copy<br />
30414.gpc-sched.OU: No such file or directory<br />
*** end error output<br />
</pre><br />
<br />
Try doing the following:<br />
<pre><br />
mkdir ${SCRATCH}/.pbs_spool<br />
ln -s ${SCRATCH}/.pbs_spool ~/.pbs_spool<br />
</pre><br />
<br />
This is how all new accounts are setup on SciNet.<br />
<br />
<tt>/home</tt> on GPC for compute jobs is mounted as a read-only file system. <br />
PBS by default tries to spool its output files to <tt>${HOME}/.pbs_spool</tt><br />
which fails as it tries to write to a read-only file <br />
system. New accounts at SciNet get around this by having ${HOME}/.pbs_spool <br />
point to somewhere appropriate on <tt>/scratch</tt>, but if you've deleted that link<br />
or directory, or had an old account, you will see errors like the above.<br />
<br />
'''On Feb 24, the input/output mechanism has been reconfigured to use a local ramdisk as the temporary location, which means that .pbs_spool is no longer needed and this error should not occur anymore.'''<br />
<br />
--><br />
<br />
=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===<br />
<br />
On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the<br />
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,<br />
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in<br />
/var/spool/torque/spool.<br />
<br />
This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.<br />
<br />
Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.<br />
<br />
=== My GPC job died, telling me `Copy Stageout Files Failed' ===<br />
<br />
'''Answer:'''<br />
<br />
When a job runs on GPC, the script's standard output and error are redirected to <br />
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in<br />
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)<br />
<br />
When you get errors like this:<br />
<pre><br />
An error has occurred processing your job, see below.<br />
request to copy stageout files failed on node<br />
</pre><br />
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.<br />
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process (e.g. ran out of memory very quickly) and those files just never got copied.<br />
<br />
Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant). <br />
<br />
Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.<br />
<br />
<!--<br />
<br />
===Another transport will be used instead===<br />
<br />
I get error messages like the following when running on the GPC at the start of the run, although the job seems to proceed OK. Is this a problem?<br />
<pre><br />
--------------------------------------------------------------------------<br />
[[45588,1],0]: A high-performance Open MPI point-to-point messaging module<br />
was unable to find any relevant network interfaces:<br />
<br />
Module: OpenFabrics (openib)<br />
Host: gpc-f101n005<br />
<br />
Another transport will be used instead, although this may result in<br />
lower performance.<br />
--------------------------------------------------------------------------<br />
</pre><br />
<br />
'''Answer:'''<br />
<br />
Everything's fine. The two MPI libraries scinet provides work for both the InifiniBand and the Gigabit Ethernet interconnects, and will always try to use the fastest interconnect available. In this case, you ran on normal gigabit GPC nodes with no infiniband; but the MPI libraries have no way of knowing this, and try the infiniband first anyway. This is just a harmless `failover' message; it tried to use the infiniband, which doesn't exist on this node, then fell back on using Gigabit ethernet (`another transport').<br />
<br />
With OpenMPI, this can be avoided by not looking for infiniband; eg, by using the option<br />
<br />
--mca btl ^openib<br />
<br />
--><br />
<br />
===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===<br />
<br />
Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.<br />
<br />
In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or <br />
network configurations may exhibit this problem at higher or lower MPI <br />
task counts.<br />
<br />
Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:<br />
<br />
<pre><br />
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288<br />
</pre><br />
<br />
With Intel MPI, you should be able to do<br />
<br />
<pre><br />
module load intelmpi/4.0.3.008<br />
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode<br />
</pre><br />
<br />
to the same end. <br />
<br />
For more information see [[GPC MPI Versions]].<br />
<br />
===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===<br />
<br />
'''Answer:'''<br />
<br />
To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.<br />
<br />
Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.<br />
<br />
--><br />
<br />
==Data on SciNet disks==<br />
<br />
===How do I find out my disk usage?===<br />
<br />
'''Answer:'''<br />
<br />
The standard unix/linux utilities for finding the amount of disk space used by a directory are very slow, and notoriously inefficient on the GPFS filesystems that we run on the SciNet systems. There are utilities that very quickly report your disk usage:<br />
<br />
The <tt>'''diskUsage'''</tt> command, available with the 'extras' module on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time.<br />
This information is only updated hourly!<br />
<br />
More information about these filesystems is available at the [[Data_Management | Data_Management]].<br />
<br />
===How do I transfer data to/from SciNet?===<br />
<br />
'''Answer:'''<br />
<br />
All incoming connections to SciNet go through relatively low-speed connections to the <tt>login.scinet</tt> gateways, so using scp to copy files the same way you ssh in is not an effective way to move lots of data. Better tools are described in our page on [[Data_Management#Data_Transfer | Data Transfer]].<br />
<br />
===My group works with data files of size 1-2 GB. Is this too large to transfer by scp to login.scinet.utoronto.ca ?===<br />
<br />
'''Answer:'''<br />
<br />
Generally, occasion transfers of data less than 10GB is perfectly acceptible to so through the login nodes. See [[Data_Management#Data_Transfer | Data Transfer]].<br />
<br />
===How can I check if I have files in /scratch that are scheduled for automatic deletion?===<br />
<br />
'''Answer:'''<br />
<br />
Please see [[Storage_Quickstart#Scratch_Disk_Purging_Policy | Storage At SciNet]]<br />
<br />
===How to allow my supervisor to manage files for me using ACL-based commands?===<br />
<br />
'''Answer:'''<br />
<br />
Please see [[Data_Management#File.2FOwnership_Management_.28ACL.29 | File/Ownership Management]]<br />
<br />
===Can we buy extra storage space on SciNet?===<br />
<br />
'''Answer:'''<br />
Yes, please see [[Data_Management#Buying_storage_space_on_GPFS_or_HPSS | Buying storage space on GPFS or HPSS ]] for more details.<br />
<br />
===Can I transfer files between BGQ and HPSS?===<br />
<br />
'''Answer:'''<br />
Yes, please see [https://support.scinet.utoronto.ca/wiki/index.php/BGQ#Bridge_to_HPSS Bridge to HPSS ] for more details.<br />
<br />
==Keep 'em Coming!==<br />
<br />
===Next question, please===<br />
<br />
Send your question to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]; we'll answer it asap!</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Knowledge_Base:_Tutorials_and_Manuals&diff=9382Knowledge Base: Tutorials and Manuals2018-08-09T17:30:23Z<p>Rzon: </p>
<hr />
<div>__NOINDEX__<br />
<br />
{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. The links below may still contain valuable information, but for current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
__TOC__<br />
<br />
=Training material=<br />
<br />
For upcoming classes, see our [https://support.scinet.utoronto.ca/education/ Training and Education website]!<br />
==SciNet Basics==<br />
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]<br />
* [[SciNet Command Line Utilities]]<br />
* Intro to SciNet May 2014: [http://support.scinet.utoronto.ca/CourseVideo/SciNetIntroMay2014/IntroMay2014.html Video]/[[Media:IntroScinetMay2014.pdf|Slides]]<br />
* SciNet Resources: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_SciNet_Resources/Monday_Morning_SciNet_Resources.mp4 Video]/ [[Media:Monday_Morning_SciNet_Resources.pdf|Slides]] <br />
* [[Essentials]]<br />
* [[FAQ|Frequenty asked questions]]<br />
* [[Ssh]]<br />
* [[GPC_Quickstart|GPC quickstart]]<br />
* [[TCS_Quickstart|TCS quickstart]]<br />
* [[GPU_Devel_Nodes|ARC/GPU quickstart]]<br />
* [[Cell_Devel_Nodes|ARC/Cell quickstart]]<br />
* [[Important .bashrc guidelines]]<br />
* [[Media:LargeScaleBio.pdf|Workflow Optimization (w/focus on Large Scale BioInformatics)]]<br />
* [[Software_and_Libraries | Software and libraries]]<br />
* [[Installing your own modules]]<br />
* [[Media:SNUGlocalsetup.pdf|User-space modules and packages (April 2011 SNUG TechTalk)]]<br />
* [[Media:HPSS_rationale.pdf|HPSS - SciNet's storage capacity expansion]]<br />
* BGQ Hardware Overview [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides ]/ [https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 Video Recording ]<br />
* Intro to Using the BGQ [[Media:Bgqintro.pdf|Slides ]]/[https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 Video Recording ]<br />
<br />
==Linux==<br />
* [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html Linux Tutorial] (warning: the last part of this tutorial is specific to the csh shell, whereas SciNet uses the bash shell).<br />
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]<br />
* Introduction to the Linux Shell, SciNet, Mar 2012: [[Media:IntroToShell.pdf|Slides]] and [[Media:Shell-data.tgz|Data files]]<br />
<br />
==Batch job management==<br />
* [https://support.scinet.utoronto.ca/education/go.php/183/file_storage/index.php/download/1/files%5B%5D/3702/ Job and Queue Management Tools] (Techtalk Mar.11, 2015)<br />
* [[Media:LargeScaleBio.pdf|Workflow Optimization (w/focus on Large Scale BioInformatics)]]<br />
* [[Media:Tech-talk-gnu-parallel.pdf|GNU Parallel (Techtalk Nov 14, 2012)]]<br />
* [[Media:TechTalkJobMonitoring.pdf|Job Monitoring on SciNet and Job Efficiency]]<br />
<!-- * [[Media:Snugtrackjob.pdf|Job Monitoring on SciNet and Job Efficiency]] --><br />
* [[Wallclock time]]<br />
* [[Checkpoints]]<br />
* [[Using_Signals|Signals]]<br />
* [[Moab]]<br />
* [[User_Serial|Serial Jobs (including GNU Parallel)]]<br />
* [[User_Ramdisk|Ramdisk]]<br />
* [http://www.clusterresources.com/products/mwm/docs/index.shtml Moab workload manager]<br />
* [http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml Moab commands]<br />
* [http://www.clusterresources.com/products/torque/docs/ Torque resource manager] <br />
* [http://www.clusterresources.com/products/torque/docs/a.acommands.shtml Torque PBS commands]<br />
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]<br />
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)<br />
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]<br />
<br />
==Programming==<br />
===General===<br />
* [[Media:SciDev-XLCompilers.pdf|Performance Tuning with the IBM XL Compilers]]: Slides fromt the SciNet Develop Seminar by Kit Barton, Sep 17, 2012.<br />
* [[Media:Remotescinet.pdf|Remote Development]], slides from TechTalk Jun 13, 2012<br />
* [[Scientific Software Development Course]], part I of the SciNet's Scientific Computing Course<br />
* [http://software-carpentry.org Software Carpentry Resources]<br />
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]<br />
* [[IBM_Nov_Workshop | IBM AIX Workshop, SciNet, Nov 2008 ]] <br />
* [[IBM_Compiler_Workshop | IBM Compiler Workshop, SciNet, Feb 2009]]<br />
* SNUG Techtalk Dec 2011 [[Media:Snug_techtalk_compiler.pdf | Intel Compiler Optimizations]]<br />
<br />
===Fortran===<br />
* Modern Fortran Course (1 day), SciNet, 19 Apr 2011<br />
** [[Media:ModernFortran.pdf | Slides]]<br />
** [[Media:ModernFortran.tgz | Source Code]]<br />
* [http://software.intel.com/sites/products/documentation/hpc/compilerp* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf IBM Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]<br />
<br />
===C++===<br />
* [[Media:Cpp11.pdf|Slides]] and [http://support.scinet.utoronto.ca/CourseVideo/Cpp11/cpp11.html recording] of the SciNet Developer Seminar on C++11, March 20, 2013<br />
* Scientific C++ Course (1 day), SciNet, 15 March 2011 <br />
** [[Media:Scientific-c%2B%2B.pdf|Slides]] (updated on Apr 26, 2012)<br />
** [[Media:Scinetcppexamples.tgz|Example source code]]<br />
** [[Videos_of_the_One-Day_Scientific_C%2B%2B_Class | Videos of the Scientific C++ class]] <br />
* [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/cpp/lin/compiler_c/index.htm Intel C & C++ compiler]<br />
ro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran compiler]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf IBM C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]<br />
<br />
===C===<br />
* C refresher: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_C_Review/Monday_Morning_C_Review.mp4 Video]/ [[Media:Monday_Morning_C_Review.pdf| Slides]]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf IBM C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]<br />
<br />
===Cilk Plus===<br />
* [[Media:TechTalkSzalwinskiCilkPlus.pdf|CilkPlus - Getting Started]] (slides of the May 2014 TechTalk by Chris Szalwinski)<br />
<br />
===Hadoop===<br />
* Introduction to Hadoop for HPCers, Part I - MapReduce: [[Media:Hadoop-PartI.pdf|Slides]], [[Media:HadoopPart1examples.tgz|Source Code]], [http://support.scinet.utoronto.ca/~ljdursi/SciNetHadoopVM.zip Virtual Machine]<br />
<br />
===Perl===<br />
* [[Perl]]<br />
===Python===<br />
* [[Python]]<br />
* [[IPython Notebook on GPC]] (January 2014 TechTalk)<br />
* [[Research Computing with Python]] (Modular Course, Fall 2013)<br />
* [[Intro to Tkinter|Python GUIs with Python and TkInter]] [[Media:Tkinter.pdf|slides]] [[Media:Tkinter_code.tgz|code]]<br />
* [http://support.scinet.utoronto.ca/Snug/scinet-f2py/scinet-f2py.html f2py: Fortran and Python] (June 2011 TechTalk by Pierre de Buyl)<br />
<br />
===R===<br />
* [[R Statistical Package]]<br />
===Lua===<br />
* [[Media:PeterColberg_Lua_scinet.pdf | Scripting HALMD with Lua and Luabind]] (May 2011 TechTalk by Peter Colberg)<br />
<br />
==Parallel Programming==<br />
* [[2014 Ontario Summerschool on High Performance Computing Central]] (incl. OpenMP and MPI)<br />
* [[High Performance Scientific Computing Course 2014]] (OpenMP and MPI, with video recordings of lectures)<br />
* [[Media:SciNet_MPI3.0_seminar.pdf | MPI 3 Developer Seminar]]<br />
* [[Media:Ds-openmp.pdf| OpenMP 4 Developer Seminar]]<br />
* [[Ontario Summerschool on High Performance Computing Central]] (2013)<br />
* [[High Performance Scientific Computing]], part 3 of SciNet's Scientific Computing Course (Winter 2012)<br />
* Parallel Programming Course (5 days), SciNet, May 2011<br />
** [[Parallel_Scientific_Computing_-_May_2011 | Videos, slides and code]]<br />
* Parallel Computing for Computational Fluid Dynamics (CFD), SciNet, 23 March 2011<br />
** [[Media:parCFD-mpi.pdf | Slides]]<br />
** [[Media:parCFD.tgz | Source Code]]<br />
* Intro to Practical Parallel Programming (1 day), SciNet, 22 Sept 2010: <br />
**[[Media:PPP-Intro-Morning.pdf|Morning Slides, Intro and OpenMP ]]<br />
**[[Media:PPP-Intro-Afternoon.pdf|Afternoon Slides, MPI]]<br />
**[[Media:Intro-ppp.tgz|Example source code]]<br />
* Parallel Scientific Computing Workshop (5 days), SciNet, Aug 2009: <br />
**[[ Parallel_Scientific_Computing_-_Aug_09 | Slides ]]<br />
**[http://www.cita.utoronto.ca/~ljdursi/PSP/ Video]<br />
* [http://www.vscse.org/ Virtual School for CSE] Web courses (Jul/Aug 2010):<br />
** Petascale programming environments and tools<br />
** Big data for science<br />
** Proven algorithmic techniques for many-core processors<br />
* [https://computing.llnl.gov/tutorials/mpi/ LLNL MPI Tutorial]: This was the basis for the MPI workshop at SciNet. <br />
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]<br />
* [[GPC MPI Versions]]<br />
* [[Co-array Fortran on the GPC]]<br />
* [[IBM_Feb_Workshop | IBM MPI Workshop, SciNet, Feb 2009]]<br />
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf IBM UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]<br />
<br />
==GPU Computing==<br />
* 1-day "Introduction to GPGPU Programming with CUDA" Course (May 2015): [https://support.scinet.utoronto.ca/education/go.php/251/index.php/ib/1//p_course/251].<br />
* 2-day "Programming GPUs with CUDA" Course (2015 Summer School): [https://wiki.scinet.utoronto.ca/wiki/index.php/2015_Ontario_Summer_School_for_High_Performance_Computing_Central#Programming_GPUs_with_CUDA]<br />
* [[Media:SNUG_NOV_CUDA.pdf | New Features in CUDA 5 & 6, Nov. 2014]]<br />
* [[Media:Westgrid_CUDA.pdf | Intro to GPU Computing Using CUDA]] (WestGrid Spring 2014 Seminar Series)<br />
* 1.5 hour intro to CUDA, March 2013: [[Media:CUDA-Graphics-Intro-2013.pdf | Slides]] and [[Media:CUDA-Graphics-Intro-2013.tgz | Source Code]]<br />
* [[CUDA_Minicourse_Fall_2012 | CITA/SciNet CUDA Minicourse, Fall 2012]]<br />
* [[SciNet GPU Workshop July 2010]]<br />
* Intro to GPGPU Programming: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_GPGPU/Friday_Morning_GPGPU.mp4 Video]/ [[Media:Gpgpu.pdf | Slides]]<br />(from 5 day parallel programming course at SciNet, May 2011)<br />
* 1-day intro to GPGPU using CUDA Course (Aug 2011): [[Media:Intro-gpu.tgz | Source Code]], [[Media:IntroGPGPU-Aug2011.pdf | Slides]].<br />
* [http://developer.nvidia.com/object/cuda_training.html NVidia archived courses for GPGPU Programming]<br />
* [http://www.pgroup.com/doc/pgiug.pdf PGI Compiler User's Guide]<br />
* [http://www.pgroup.com/doc/pgiref.pdf PGI Compiler Reference Manual]<br />
* [http://www.pgroup.com/doc/pgifortref.pdf PGI Fortran reference]<br />
* [http://www.pgroup.com/doc/pgicudaforug.pdf PGI CUDA Fortran Programming Guide and Reference]<br />
* [http://www.pgroup.com/doc/openACC_gs.pdf PGI OpenACC Getting Started Guide]<br />
<br />
==Performance Tuning==<br />
* [[Performance and Profiling Course, April 2013]]<br />
* [[Introduction To Performance]]<br />
* Performance tools for [[Performance_And_Debugging_Tools:_GPC | GPC ]] and [[Performance_And_Debugging_Tools:_TCS | TCS ]]<br />
* Dec 2010 SNUG TechTalk: [[Media:ProfillingTechTalk-Dec2010.pdf | Profiling Tools on GPC]]<br />
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]<br />
* [[Media:Mpi-tuning-parameters.pdf | MPI Tuning Parameters]] - SNUG TechTalk, Feb 2012<br />
* [http://cnx.org/content/col11136/latest/ High Performance Computing Book]<br />Online version of an older O'Reilly book which covers the basics of (mostly serial) programming for performance. Covers the most important issues today (such as cache) very clearly.<br />
* [http://www.ece.cmu.edu/~franzf/papers/gttse07.pdf How to Write Fast Numerical Code ]<br />Good introduction to thinking about performance and cache.<br />
<br />
==Debugging==<br />
* [[Parallel Debugging with DDT]]<br />
* [[Media:SS_Debug.pdf|Debugging with GDB and DDT, half-day session at the Ontario HPC Summerschool 2012 Central<br>Slides]], [[Media:SS_Debug.tgz|Code]].<br />
* [[Media:Snugdebug.pdf|TechTalk: Debuggers & Parallel Debugging on SciNet - gdb, ddd, padb], SciNet User Group Meeting, Nov 2010]]<br/ > [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_Debugging/Thursday_Morning_Debugging.mp4 Video]<br />
* [http://www.allinea.com/downloads/userguide.pdf Allinea DDT (Distributed Debugging Tool) User Guide]<br />
<br />
==Math libraries (BLAS, LAPACK, FFT)==<br />
* [[Media:MKLTechTalkMarch2012.pdf|Intel Math Kernel Library (MKL): An overview]] (TechTalk, March, 2012)<br />
* [[Numerical Tools for Physical Scientists]], part 2 of SciNet's Scientific computing course, covers, random nubers, blas, lapack, fft, ...<br />
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]<br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)] <br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]<br />
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]<br/>($MKLPATH &rarr; ${MKLPATH} in makefiles)<br />
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])<br />
* [http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf Parallel ESSL high performance math library V3.3]<br />
* [http://hal.inria.fr/inria-00576469 Linear Algebra Libraries] by Claire Mouton. 2009 INRIA Technical Report on existing linear algebra libraries for C++ (also here: [http://arxiv.org/abs/1103.3020])<br />
<br />
==I/O==<br />
<br />
* [[Media:NetCDF.pdf|Introduction to NetCDF4 binary files with Python, C++ and R (TechTalk March 2014)]]<br />
* [[Using MySQL on the GPC]]<br />
* [https://support.scinet.utoronto.ca/education/go.php/19/content.php/cid/59/ Relational Database Basics]<br />
* [[Media:SCIENCEDATA.pdf|Sep 2012 SNUG TechTalk: Science=Data]]<br />
* [[Data_Management|Data management]]<br />
* Intro to Parallel I/O, SciNet, Oct 6th, 2010: <br />
**[[Media:Parallel_io_course.pdf|Morning & MPI-IO Slides]]<br />
**[[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]<br />
**[[Media:ParIO.tgz|Source Code]]. <br />
* Intro to Parallel I/O, SciNet, Feb, 2013 <br />
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-intro/pario-intro.mp4 Video of the Introduction]<br />
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-netcdfhdf5/pario-netcdfhdf5.mp4 Video of the parallel hdf5 and netcdf part] [[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]<br />
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-mpiio/pario-mpiio.mp4 Video of the MPI-IO part]<br />
* Half-day HPCS2012 Parallel I/O tutorial, covering MPI-IO, HDF5, NetCDF, based on the above: [[Media:ParIO-HPCS2012.pdf|slides (pdf)]] and [[Media:ParIO-HPCS2012.tgz|source code]].<br />
* [[Media:Snugio.pdf|Sept 2010 SNUG TechTalk: Parallel File System and IO]] <br/ >[http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_IO/Friday_Morning_IO.mp4 Video]<br />
* [[File System and I/O dos and don'ts]]<br />
* [[Media:40TB.pdf|So you have 40TB of Data]] -- an overview of things to consider with large data sets.<br />
* [[Media:Adios-techtalk-may2012.pdf|May 2012 SNUG TechTalk: ADIOS for Parallel IO slides]] and [[Media:Adios-techtalk-may2012-src.tgz|source code]]<br />
* [[hdf5_table|Writting / Reading a table in hdf5]]<br />
* [[NetCDF_table|Writting / Reading a table in NetCDF]]<br />
* Intro to Parallel I/O, SciNet, Sept 215<br />
** MPI-IO [[Media:ParIO.MPIIO.2015.pdf |Slides]] and [[Media:ParIO.MPIIO.tar.gz | Sample codes]]<br />
** [[Media:netcdf2015.pdf | NETCDF]] & [[Media:hfd5.pdf | HDF5]]<br />
<br />
==Infiniband Networking==<br />
* [[Media:Snug_techtalk_Infiniband.pdf | TechTalk on SciNet's Infiniband Network & MPI options ]] <br />
<br />
==Visualization==<br />
* [[Using Paraview]]<br />
* [[VNC|VNC on the GPC]] (see also [[Media:Ttvnc.pdf|slides of the TechTalk on VNC]])<br />
* [[Software_and_Libraries#anchor_viz|Visualization Software on the GPC]]<br />
* [https://wiki.scinet.utoronto.ca/wiki/images/5/51/Remoteviz.pdf Remote visualization (X-forwarding and VNC)]<br />
* [[Using Paraview]]<br />
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-I.pdf gnuplot, xmgrace, remote visualization tools (X-forwarding and VNC), python's matplotlib]<br />
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-II.pdf Brief overview of ParaView & VisIt]<br />
* [https://support.scinet.utoronto.ca/education/go.php/242/file_storage/index.php/download/1/files%5B%5D/6399/ VisIt Basics]<br />
* [https://support.scinet.utoronto.ca/education/go.php/273/file_storage/index.php/download/1/files%5B%5D/7363/ Scientific Visualization using VisIt (Basics and Advanced)], M.Ponce and A.Razoumov<br />
* [https://support.scinet.utoronto.ca/education/get.php/8_ComplexNetworks.pdf Intro to Complex Networks Visualization, with Python]<br />
<br />
<br />
* [http://scienceillustrated.ca Science Illustrated:] Two-day symposium on Visualizing Science, Feb 2011<br />
* [http://www.kmdi.utoronto.ca/story/2011/03/si-science-illustrated-symposium-success Videos of the talks given at Science Illustrated] (recorded by [http://www.kmdi.utoronto.ca KMDI] at [http://www.utoronto.ca UoT]):<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=94ff5cd5-be6e-4fc6-9be6-dd2222342bcd Opening remarks] by Paul Young<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4255c34e-15e7-4b24-ba99-78f5c8fa4381 Information Visualization and the Myth of Information Overload] by Christopher Collins<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=adcf02bf-16cb-46cc-8cdc-1a65e9071d6b Beyond Basic Visualization] by Ramses Van Zon<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=47baf346-3599-4fa3-9b10-6a58faa6b33c Network Visualization &amp; Analysis] by Igor Jurisica<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7e754a2e-7be5-476e-bb54-37def37bc07e Simulation and Visualization of Blood Flow] by David Steinman<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=d7622587-2c31-49c1-99d7-f0f16c078801 Scientific Visualizations: Does the Science Matter?] by Thomas Lucas<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=9963c637-6840-454f-a57f-9a1be6456616 How can visualization impact public perception of science?] Panelists: Jay Ingram, Peter Calamai, Reni Barlow, Hooley McLaughlin<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=88e50cff-db9b-4c71-b10d-781fec60a2c0 How Info Graphics are Created for the Mainstream Media] by Peter Calamai<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=3897e2a3-1fda-42be-ab78-edbab090fd9e Design Boot Camp] by Graham Huber<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4c27ed76-7292-407e-83a6-814e1461eccd Visualization Large Datasets] by Jonathan Dursi<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d49c845-3937-44e2-a300-9b8ffe57a857 Visualizing Colliding Black Holes] by Herald Pfeiffer<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d7e4803-39cd-4c8e-a443-bc2a7b1b3c28 Closing remarks] by Mubdi Rahman<br />
<br />
==Applications==<br />
{{:Knowledge Base: Applications}}<br />
* See also [[User Codes]]<br />
<br />
== Development Systems ==<br />
<br />
=== KNL ===<br />
<br />
* SNUG TechTalk October 2016 - [[Media:Snug_techtalk_KNL.pdf | Intel Xeon Phi Knights Landing ]]<br />
<br />
=Manuals=<br />
<br />
==Intel compilers and libraries (GPC)==<br />
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-lin/index.htm C & C++ compiler]<br />
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/fortran-lin/index.htm Fortran compiler]<br />
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]<br />
* [[Media:Compiler_qrg12.pdf|Intel Compiler Optimization Guide]]<br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)] <br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]<br />
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]<br />
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]<br/>($MKLPATH &rarr; ${MKLPATH} in makefiles)<br />
<br />
==IBM compilers and libraries (TCS/P7)==<br />
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]<br />
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]<br />
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])<br />
* [[Media:essl51.pdf|ESSL high performance math library V5.1 for Linux on Power]]<br />
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]<br />
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]<br />
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)<br />
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]<br />
<br />
==PGI compilers (ARC)==<br />
* [http://www.pgroup.com/doc/pgiug.pdf Compiler User's Guide]<br />
* [http://www.pgroup.com/doc/pgiref.pdf Compiler Reference Manual]<br />
* [http://www.pgroup.com/doc/pgifortref.pdf Fortran reference]<br />
* [http://www.pgroup.com/doc/pgicudaforug.pdf CUDA Fortran Programming Guide and Reference]<br />
* [http://www.pgroup.com/doc/openACC_gs.pdf OpenACC Getting Started Guide]<br>(Note: $PGI/linux86-64/12.5/doc contains a newer version.)<br />
<br />
==Scheduler (Adaptive Computing/Cluster Resources)==<br />
* [http://docs.adaptivecomputing.com/mwm/archive/7-0/help.htm Moab workload manager]<br />
* [http://docs.adaptivecomputing.com/mwm/archive/7-0/Content/a.gcommandoverview.html Moab commands]<br />
* [http://docs.adaptivecomputing.com/torque/archive/4-0-1/help.htm Torque resource manager] <br />
* [http://docs.adaptivecomputing.com/torque/archive/4-0-1/help.htm#topics/12-appendices/commandsOverview.htm Torque PBS commands]<br />
<br />
==DDT Debugger (Allinea)==<br />
* [http://www.allinea.com/downloads/userguide.pdf Distributed Debugging Tool User Guide]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Knowledge_Base:_Tutorials_and_Manuals&diff=9381Knowledge Base: Tutorials and Manuals2018-08-09T17:30:10Z<p>Rzon: </p>
<hr />
<div>__NOINDEX__<br />
__TOC__<br />
<br />
{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. The links below may still contain valuable information, but for current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
=Training material=<br />
<br />
For upcoming classes, see our [https://support.scinet.utoronto.ca/education/ Training and Education website]!<br />
==SciNet Basics==<br />
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]<br />
* [[SciNet Command Line Utilities]]<br />
* Intro to SciNet May 2014: [http://support.scinet.utoronto.ca/CourseVideo/SciNetIntroMay2014/IntroMay2014.html Video]/[[Media:IntroScinetMay2014.pdf|Slides]]<br />
* SciNet Resources: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_SciNet_Resources/Monday_Morning_SciNet_Resources.mp4 Video]/ [[Media:Monday_Morning_SciNet_Resources.pdf|Slides]] <br />
* [[Essentials]]<br />
* [[FAQ|Frequenty asked questions]]<br />
* [[Ssh]]<br />
* [[GPC_Quickstart|GPC quickstart]]<br />
* [[TCS_Quickstart|TCS quickstart]]<br />
* [[GPU_Devel_Nodes|ARC/GPU quickstart]]<br />
* [[Cell_Devel_Nodes|ARC/Cell quickstart]]<br />
* [[Important .bashrc guidelines]]<br />
* [[Media:LargeScaleBio.pdf|Workflow Optimization (w/focus on Large Scale BioInformatics)]]<br />
* [[Software_and_Libraries | Software and libraries]]<br />
* [[Installing your own modules]]<br />
* [[Media:SNUGlocalsetup.pdf|User-space modules and packages (April 2011 SNUG TechTalk)]]<br />
* [[Media:HPSS_rationale.pdf|HPSS - SciNet's storage capacity expansion]]<br />
* BGQ Hardware Overview [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides ]/ [https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 Video Recording ]<br />
* Intro to Using the BGQ [[Media:Bgqintro.pdf|Slides ]]/[https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 Video Recording ]<br />
<br />
==Linux==<br />
* [http://www.ee.surrey.ac.uk/Teaching/Unix/index.html Linux Tutorial] (warning: the last part of this tutorial is specific to the csh shell, whereas SciNet uses the bash shell).<br />
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]<br />
* Introduction to the Linux Shell, SciNet, Mar 2012: [[Media:IntroToShell.pdf|Slides]] and [[Media:Shell-data.tgz|Data files]]<br />
<br />
==Batch job management==<br />
* [https://support.scinet.utoronto.ca/education/go.php/183/file_storage/index.php/download/1/files%5B%5D/3702/ Job and Queue Management Tools] (Techtalk Mar.11, 2015)<br />
* [[Media:LargeScaleBio.pdf|Workflow Optimization (w/focus on Large Scale BioInformatics)]]<br />
* [[Media:Tech-talk-gnu-parallel.pdf|GNU Parallel (Techtalk Nov 14, 2012)]]<br />
* [[Media:TechTalkJobMonitoring.pdf|Job Monitoring on SciNet and Job Efficiency]]<br />
<!-- * [[Media:Snugtrackjob.pdf|Job Monitoring on SciNet and Job Efficiency]] --><br />
* [[Wallclock time]]<br />
* [[Checkpoints]]<br />
* [[Using_Signals|Signals]]<br />
* [[Moab]]<br />
* [[User_Serial|Serial Jobs (including GNU Parallel)]]<br />
* [[User_Ramdisk|Ramdisk]]<br />
* [http://www.clusterresources.com/products/mwm/docs/index.shtml Moab workload manager]<br />
* [http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml Moab commands]<br />
* [http://www.clusterresources.com/products/torque/docs/ Torque resource manager] <br />
* [http://www.clusterresources.com/products/torque/docs/a.acommands.shtml Torque PBS commands]<br />
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]<br />
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)<br />
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]<br />
<br />
==Programming==<br />
===General===<br />
* [[Media:SciDev-XLCompilers.pdf|Performance Tuning with the IBM XL Compilers]]: Slides fromt the SciNet Develop Seminar by Kit Barton, Sep 17, 2012.<br />
* [[Media:Remotescinet.pdf|Remote Development]], slides from TechTalk Jun 13, 2012<br />
* [[Scientific Software Development Course]], part I of the SciNet's Scientific Computing Course<br />
* [http://software-carpentry.org Software Carpentry Resources]<br />
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]<br />
* [[IBM_Nov_Workshop | IBM AIX Workshop, SciNet, Nov 2008 ]] <br />
* [[IBM_Compiler_Workshop | IBM Compiler Workshop, SciNet, Feb 2009]]<br />
* SNUG Techtalk Dec 2011 [[Media:Snug_techtalk_compiler.pdf | Intel Compiler Optimizations]]<br />
<br />
===Fortran===<br />
* Modern Fortran Course (1 day), SciNet, 19 Apr 2011<br />
** [[Media:ModernFortran.pdf | Slides]]<br />
** [[Media:ModernFortran.tgz | Source Code]]<br />
* [http://software.intel.com/sites/products/documentation/hpc/compilerp* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf IBM Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]<br />
<br />
===C++===<br />
* [[Media:Cpp11.pdf|Slides]] and [http://support.scinet.utoronto.ca/CourseVideo/Cpp11/cpp11.html recording] of the SciNet Developer Seminar on C++11, March 20, 2013<br />
* Scientific C++ Course (1 day), SciNet, 15 March 2011 <br />
** [[Media:Scientific-c%2B%2B.pdf|Slides]] (updated on Apr 26, 2012)<br />
** [[Media:Scinetcppexamples.tgz|Example source code]]<br />
** [[Videos_of_the_One-Day_Scientific_C%2B%2B_Class | Videos of the Scientific C++ class]] <br />
* [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/cpp/lin/compiler_c/index.htm Intel C & C++ compiler]<br />
ro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran compiler]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf IBM C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]<br />
<br />
===C===<br />
* C refresher: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_C_Review/Monday_Morning_C_Review.mp4 Video]/ [[Media:Monday_Morning_C_Review.pdf| Slides]]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf IBM C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]<br />
<br />
===Cilk Plus===<br />
* [[Media:TechTalkSzalwinskiCilkPlus.pdf|CilkPlus - Getting Started]] (slides of the May 2014 TechTalk by Chris Szalwinski)<br />
<br />
===Hadoop===<br />
* Introduction to Hadoop for HPCers, Part I - MapReduce: [[Media:Hadoop-PartI.pdf|Slides]], [[Media:HadoopPart1examples.tgz|Source Code]], [http://support.scinet.utoronto.ca/~ljdursi/SciNetHadoopVM.zip Virtual Machine]<br />
<br />
===Perl===<br />
* [[Perl]]<br />
===Python===<br />
* [[Python]]<br />
* [[IPython Notebook on GPC]] (January 2014 TechTalk)<br />
* [[Research Computing with Python]] (Modular Course, Fall 2013)<br />
* [[Intro to Tkinter|Python GUIs with Python and TkInter]] [[Media:Tkinter.pdf|slides]] [[Media:Tkinter_code.tgz|code]]<br />
* [http://support.scinet.utoronto.ca/Snug/scinet-f2py/scinet-f2py.html f2py: Fortran and Python] (June 2011 TechTalk by Pierre de Buyl)<br />
<br />
===R===<br />
* [[R Statistical Package]]<br />
===Lua===<br />
* [[Media:PeterColberg_Lua_scinet.pdf | Scripting HALMD with Lua and Luabind]] (May 2011 TechTalk by Peter Colberg)<br />
<br />
==Parallel Programming==<br />
* [[2014 Ontario Summerschool on High Performance Computing Central]] (incl. OpenMP and MPI)<br />
* [[High Performance Scientific Computing Course 2014]] (OpenMP and MPI, with video recordings of lectures)<br />
* [[Media:SciNet_MPI3.0_seminar.pdf | MPI 3 Developer Seminar]]<br />
* [[Media:Ds-openmp.pdf| OpenMP 4 Developer Seminar]]<br />
* [[Ontario Summerschool on High Performance Computing Central]] (2013)<br />
* [[High Performance Scientific Computing]], part 3 of SciNet's Scientific Computing Course (Winter 2012)<br />
* Parallel Programming Course (5 days), SciNet, May 2011<br />
** [[Parallel_Scientific_Computing_-_May_2011 | Videos, slides and code]]<br />
* Parallel Computing for Computational Fluid Dynamics (CFD), SciNet, 23 March 2011<br />
** [[Media:parCFD-mpi.pdf | Slides]]<br />
** [[Media:parCFD.tgz | Source Code]]<br />
* Intro to Practical Parallel Programming (1 day), SciNet, 22 Sept 2010: <br />
**[[Media:PPP-Intro-Morning.pdf|Morning Slides, Intro and OpenMP ]]<br />
**[[Media:PPP-Intro-Afternoon.pdf|Afternoon Slides, MPI]]<br />
**[[Media:Intro-ppp.tgz|Example source code]]<br />
* Parallel Scientific Computing Workshop (5 days), SciNet, Aug 2009: <br />
**[[ Parallel_Scientific_Computing_-_Aug_09 | Slides ]]<br />
**[http://www.cita.utoronto.ca/~ljdursi/PSP/ Video]<br />
* [http://www.vscse.org/ Virtual School for CSE] Web courses (Jul/Aug 2010):<br />
** Petascale programming environments and tools<br />
** Big data for science<br />
** Proven algorithmic techniques for many-core processors<br />
* [https://computing.llnl.gov/tutorials/mpi/ LLNL MPI Tutorial]: This was the basis for the MPI workshop at SciNet. <br />
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]<br />
* [[GPC MPI Versions]]<br />
* [[Co-array Fortran on the GPC]]<br />
* [[IBM_Feb_Workshop | IBM MPI Workshop, SciNet, Feb 2009]]<br />
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf IBM UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]<br />
<br />
==GPU Computing==<br />
* 1-day "Introduction to GPGPU Programming with CUDA" Course (May 2015): [https://support.scinet.utoronto.ca/education/go.php/251/index.php/ib/1//p_course/251].<br />
* 2-day "Programming GPUs with CUDA" Course (2015 Summer School): [https://wiki.scinet.utoronto.ca/wiki/index.php/2015_Ontario_Summer_School_for_High_Performance_Computing_Central#Programming_GPUs_with_CUDA]<br />
* [[Media:SNUG_NOV_CUDA.pdf | New Features in CUDA 5 & 6, Nov. 2014]]<br />
* [[Media:Westgrid_CUDA.pdf | Intro to GPU Computing Using CUDA]] (WestGrid Spring 2014 Seminar Series)<br />
* 1.5 hour intro to CUDA, March 2013: [[Media:CUDA-Graphics-Intro-2013.pdf | Slides]] and [[Media:CUDA-Graphics-Intro-2013.tgz | Source Code]]<br />
* [[CUDA_Minicourse_Fall_2012 | CITA/SciNet CUDA Minicourse, Fall 2012]]<br />
* [[SciNet GPU Workshop July 2010]]<br />
* Intro to GPGPU Programming: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_GPGPU/Friday_Morning_GPGPU.mp4 Video]/ [[Media:Gpgpu.pdf | Slides]]<br />(from 5 day parallel programming course at SciNet, May 2011)<br />
* 1-day intro to GPGPU using CUDA Course (Aug 2011): [[Media:Intro-gpu.tgz | Source Code]], [[Media:IntroGPGPU-Aug2011.pdf | Slides]].<br />
* [http://developer.nvidia.com/object/cuda_training.html NVidia archived courses for GPGPU Programming]<br />
* [http://www.pgroup.com/doc/pgiug.pdf PGI Compiler User's Guide]<br />
* [http://www.pgroup.com/doc/pgiref.pdf PGI Compiler Reference Manual]<br />
* [http://www.pgroup.com/doc/pgifortref.pdf PGI Fortran reference]<br />
* [http://www.pgroup.com/doc/pgicudaforug.pdf PGI CUDA Fortran Programming Guide and Reference]<br />
* [http://www.pgroup.com/doc/openACC_gs.pdf PGI OpenACC Getting Started Guide]<br />
<br />
==Performance Tuning==<br />
* [[Performance and Profiling Course, April 2013]]<br />
* [[Introduction To Performance]]<br />
* Performance tools for [[Performance_And_Debugging_Tools:_GPC | GPC ]] and [[Performance_And_Debugging_Tools:_TCS | TCS ]]<br />
* Dec 2010 SNUG TechTalk: [[Media:ProfillingTechTalk-Dec2010.pdf | Profiling Tools on GPC]]<br />
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]<br />
* [[Media:Mpi-tuning-parameters.pdf | MPI Tuning Parameters]] - SNUG TechTalk, Feb 2012<br />
* [http://cnx.org/content/col11136/latest/ High Performance Computing Book]<br />Online version of an older O'Reilly book which covers the basics of (mostly serial) programming for performance. Covers the most important issues today (such as cache) very clearly.<br />
* [http://www.ece.cmu.edu/~franzf/papers/gttse07.pdf How to Write Fast Numerical Code ]<br />Good introduction to thinking about performance and cache.<br />
<br />
==Debugging==<br />
* [[Parallel Debugging with DDT]]<br />
* [[Media:SS_Debug.pdf|Debugging with GDB and DDT, half-day session at the Ontario HPC Summerschool 2012 Central<br>Slides]], [[Media:SS_Debug.tgz|Code]].<br />
* [[Media:Snugdebug.pdf|TechTalk: Debuggers & Parallel Debugging on SciNet - gdb, ddd, padb], SciNet User Group Meeting, Nov 2010]]<br/ > [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_Debugging/Thursday_Morning_Debugging.mp4 Video]<br />
* [http://www.allinea.com/downloads/userguide.pdf Allinea DDT (Distributed Debugging Tool) User Guide]<br />
<br />
==Math libraries (BLAS, LAPACK, FFT)==<br />
* [[Media:MKLTechTalkMarch2012.pdf|Intel Math Kernel Library (MKL): An overview]] (TechTalk, March, 2012)<br />
* [[Numerical Tools for Physical Scientists]], part 2 of SciNet's Scientific computing course, covers, random nubers, blas, lapack, fft, ...<br />
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]<br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)] <br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]<br />
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]<br/>($MKLPATH &rarr; ${MKLPATH} in makefiles)<br />
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])<br />
* [http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf Parallel ESSL high performance math library V3.3]<br />
* [http://hal.inria.fr/inria-00576469 Linear Algebra Libraries] by Claire Mouton. 2009 INRIA Technical Report on existing linear algebra libraries for C++ (also here: [http://arxiv.org/abs/1103.3020])<br />
<br />
==I/O==<br />
<br />
* [[Media:NetCDF.pdf|Introduction to NetCDF4 binary files with Python, C++ and R (TechTalk March 2014)]]<br />
* [[Using MySQL on the GPC]]<br />
* [https://support.scinet.utoronto.ca/education/go.php/19/content.php/cid/59/ Relational Database Basics]<br />
* [[Media:SCIENCEDATA.pdf|Sep 2012 SNUG TechTalk: Science=Data]]<br />
* [[Data_Management|Data management]]<br />
* Intro to Parallel I/O, SciNet, Oct 6th, 2010: <br />
**[[Media:Parallel_io_course.pdf|Morning & MPI-IO Slides]]<br />
**[[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]<br />
**[[Media:ParIO.tgz|Source Code]]. <br />
* Intro to Parallel I/O, SciNet, Feb, 2013 <br />
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-intro/pario-intro.mp4 Video of the Introduction]<br />
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-netcdfhdf5/pario-netcdfhdf5.mp4 Video of the parallel hdf5 and netcdf part] [[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]<br />
** [http://support.scinet.utoronto.ca/CourseVideo/ParallelIOcourse/pario-mpiio/pario-mpiio.mp4 Video of the MPI-IO part]<br />
* Half-day HPCS2012 Parallel I/O tutorial, covering MPI-IO, HDF5, NetCDF, based on the above: [[Media:ParIO-HPCS2012.pdf|slides (pdf)]] and [[Media:ParIO-HPCS2012.tgz|source code]].<br />
* [[Media:Snugio.pdf|Sept 2010 SNUG TechTalk: Parallel File System and IO]] <br/ >[http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_IO/Friday_Morning_IO.mp4 Video]<br />
* [[File System and I/O dos and don'ts]]<br />
* [[Media:40TB.pdf|So you have 40TB of Data]] -- an overview of things to consider with large data sets.<br />
* [[Media:Adios-techtalk-may2012.pdf|May 2012 SNUG TechTalk: ADIOS for Parallel IO slides]] and [[Media:Adios-techtalk-may2012-src.tgz|source code]]<br />
* [[hdf5_table|Writting / Reading a table in hdf5]]<br />
* [[NetCDF_table|Writting / Reading a table in NetCDF]]<br />
* Intro to Parallel I/O, SciNet, Sept 215<br />
** MPI-IO [[Media:ParIO.MPIIO.2015.pdf |Slides]] and [[Media:ParIO.MPIIO.tar.gz | Sample codes]]<br />
** [[Media:netcdf2015.pdf | NETCDF]] & [[Media:hfd5.pdf | HDF5]]<br />
<br />
==Infiniband Networking==<br />
* [[Media:Snug_techtalk_Infiniband.pdf | TechTalk on SciNet's Infiniband Network & MPI options ]] <br />
<br />
==Visualization==<br />
* [[Using Paraview]]<br />
* [[VNC|VNC on the GPC]] (see also [[Media:Ttvnc.pdf|slides of the TechTalk on VNC]])<br />
* [[Software_and_Libraries#anchor_viz|Visualization Software on the GPC]]<br />
* [https://wiki.scinet.utoronto.ca/wiki/images/5/51/Remoteviz.pdf Remote visualization (X-forwarding and VNC)]<br />
* [[Using Paraview]]<br />
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-I.pdf gnuplot, xmgrace, remote visualization tools (X-forwarding and VNC), python's matplotlib]<br />
* [https://support.scinet.utoronto.ca/~mponce/ss2016/ss2016_visualization-II.pdf Brief overview of ParaView & VisIt]<br />
* [https://support.scinet.utoronto.ca/education/go.php/242/file_storage/index.php/download/1/files%5B%5D/6399/ VisIt Basics]<br />
* [https://support.scinet.utoronto.ca/education/go.php/273/file_storage/index.php/download/1/files%5B%5D/7363/ Scientific Visualization using VisIt (Basics and Advanced)], M.Ponce and A.Razoumov<br />
* [https://support.scinet.utoronto.ca/education/get.php/8_ComplexNetworks.pdf Intro to Complex Networks Visualization, with Python]<br />
<br />
<br />
* [http://scienceillustrated.ca Science Illustrated:] Two-day symposium on Visualizing Science, Feb 2011<br />
* [http://www.kmdi.utoronto.ca/story/2011/03/si-science-illustrated-symposium-success Videos of the talks given at Science Illustrated] (recorded by [http://www.kmdi.utoronto.ca KMDI] at [http://www.utoronto.ca UoT]):<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=94ff5cd5-be6e-4fc6-9be6-dd2222342bcd Opening remarks] by Paul Young<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4255c34e-15e7-4b24-ba99-78f5c8fa4381 Information Visualization and the Myth of Information Overload] by Christopher Collins<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=adcf02bf-16cb-46cc-8cdc-1a65e9071d6b Beyond Basic Visualization] by Ramses Van Zon<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=47baf346-3599-4fa3-9b10-6a58faa6b33c Network Visualization &amp; Analysis] by Igor Jurisica<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7e754a2e-7be5-476e-bb54-37def37bc07e Simulation and Visualization of Blood Flow] by David Steinman<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=d7622587-2c31-49c1-99d7-f0f16c078801 Scientific Visualizations: Does the Science Matter?] by Thomas Lucas<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=9963c637-6840-454f-a57f-9a1be6456616 How can visualization impact public perception of science?] Panelists: Jay Ingram, Peter Calamai, Reni Barlow, Hooley McLaughlin<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=88e50cff-db9b-4c71-b10d-781fec60a2c0 How Info Graphics are Created for the Mainstream Media] by Peter Calamai<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=3897e2a3-1fda-42be-ab78-edbab090fd9e Design Boot Camp] by Graham Huber<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4c27ed76-7292-407e-83a6-814e1461eccd Visualization Large Datasets] by Jonathan Dursi<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d49c845-3937-44e2-a300-9b8ffe57a857 Visualizing Colliding Black Holes] by Herald Pfeiffer<br />
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d7e4803-39cd-4c8e-a443-bc2a7b1b3c28 Closing remarks] by Mubdi Rahman<br />
<br />
==Applications==<br />
{{:Knowledge Base: Applications}}<br />
* See also [[User Codes]]<br />
<br />
== Development Systems ==<br />
<br />
=== KNL ===<br />
<br />
* SNUG TechTalk October 2016 - [[Media:Snug_techtalk_KNL.pdf | Intel Xeon Phi Knights Landing ]]<br />
<br />
=Manuals=<br />
<br />
==Intel compilers and libraries (GPC)==<br />
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-lin/index.htm C & C++ compiler]<br />
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/fortran-lin/index.htm Fortran compiler]<br />
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]<br />
* [[Media:Compiler_qrg12.pdf|Intel Compiler Optimization Guide]]<br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)] <br />
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]<br />
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]<br />
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor]<br/>($MKLPATH &rarr; ${MKLPATH} in makefiles)<br />
<br />
==IBM compilers and libraries (TCS/P7)==<br />
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]<br />
* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]<br />
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]<br />
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])<br />
* [[Media:essl51.pdf|ESSL high performance math library V5.1 for Linux on Power]]<br />
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]<br />
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]<br />
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)<br />
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]<br />
<br />
==PGI compilers (ARC)==<br />
* [http://www.pgroup.com/doc/pgiug.pdf Compiler User's Guide]<br />
* [http://www.pgroup.com/doc/pgiref.pdf Compiler Reference Manual]<br />
* [http://www.pgroup.com/doc/pgifortref.pdf Fortran reference]<br />
* [http://www.pgroup.com/doc/pgicudaforug.pdf CUDA Fortran Programming Guide and Reference]<br />
* [http://www.pgroup.com/doc/openACC_gs.pdf OpenACC Getting Started Guide]<br>(Note: $PGI/linux86-64/12.5/doc contains a newer version.)<br />
<br />
==Scheduler (Adaptive Computing/Cluster Resources)==<br />
* [http://docs.adaptivecomputing.com/mwm/archive/7-0/help.htm Moab workload manager]<br />
* [http://docs.adaptivecomputing.com/mwm/archive/7-0/Content/a.gcommandoverview.html Moab commands]<br />
* [http://docs.adaptivecomputing.com/torque/archive/4-0-1/help.htm Torque resource manager] <br />
* [http://docs.adaptivecomputing.com/torque/archive/4-0-1/help.htm#topics/12-appendices/commandsOverview.htm Torque PBS commands]<br />
<br />
==DDT Debugger (Allinea)==<br />
* [http://www.allinea.com/downloads/userguide.pdf Distributed Debugging Tool User Guide]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Jupyter_Hub_on_SciNet&diff=9380Jupyter Hub on SciNet2018-08-09T17:29:32Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
'''Disclaimer: the following describes is an experimental setup at SciNet running on good, but out-of-warrantee hardware.*<br />
<br />
* Four jupyterhub servers, each with 128 GB of memory and 16 or 20 cores.<br />
* Access using an ssh tunnel via login.scinet.utoronto.ca:<br />
<br />
$ ssh USER@login.scinet.utoronto.ca -L8888:jupyterhub:8000 -N -f<br />
<br />
* This will select (round-robin) one of the four jupyterhub servers.<br />
* Point your browser to 'localhost:8888' and log in with your SciNet account.<br />
* The browser should now show the files in your $HOME on SciNet. (If not, try reloading the page, it may have timed out).<br />
* You can open or create Python 2, Python 3, and R notebooks.<br />
* Large number of python packages preinstalled.<br />
<br />
[[File:jupyterscreen3.png]]<br />
<br />
===Tips to get started===<br />
<br />
* Jupyter can also browse your (SciNet) files and edit them.<br />
* Use the 'new' button to create a new python notebook.<br />
* Give your notebooks reasonable names.<br />
* To execute a python input line, press `Shift-Enter`.<br />
* Save your work periodically (even though there is autosave).<br />
* To work similarly to `ipython --pylab`, do:<br />
<br />
In [1]: from pylab import *<br />
%matplotlib notebook<br />
<br />
===Advantages and Disadvantages of a Notebook Environment===<br />
<br />
Drawbacks:<br />
* Notebook files (.ipynb) are not scripts.<br />
* Does not (always) work well with version control.<br />
* Designed to run in browser.<br />
* Back-end runs on shared resources.<br />
* Graphics is inline, which is great for quick exploration but make tweaking a plot harder (IPython+X works better for this).<br />
* You can jump around in the notebook, and execute different parts: hard to keep track of what you did.<br />
<br />
Advantages:<br />
* You can jump around in the notebook, and execute different parts: Easier exploration, experimentation and debugging.<br />
* Auto-save<br />
* You can rerun parts of your code (while, e.g., keeping large data in memory)<br />
* You can add text portions, making your notebook more like an article.<br />
* Which in turn can be useful for sharing, demos, teaching, ...<br />
* You can still export as a script.<br />
* Also has a terminal.</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=Knights_Landing&diff=9379Knights Landing2018-08-09T17:29:19Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:KNL-DAP-Adams-Pass.jpg|center|250px|thumb]]<br />
|name=Intel Xeon Phi (Knights Landing )<br />
|installed=August 2016<br />
|operatingsystem= Linux Centos 7.2<br />
|loginnode= knl01<br />
|nnodes=4<br />
|rampernode=96GB DDR4 + 16GB MCDRAM<br />
|corespernode=64<br />
|interconnect=QDR Infiniband<br />
|vendorcompilers=icc,ifort<br />
|queuetype=none<br />
}}<br />
<br />
This is develop/test system of four x86_64 self-hosted 2nd Generation Intel Xeon Phi (Knights Landing, KNL) nodes, aka an Intel "[http://dap.xeonphi.com/#platformspecs Ninja]" platform. Each node has one 64-core Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz with 4 threads per core. These systems are not add-on accelerators, but instead act as full-fledged processors running a regular linux operating system. They are configured with 96GB of DDR4 system RAM along with 16GB of very fast MCDRAM, see [https://software.intel.com/en-us/blogs/2016/01/20/an-intro-to-mcdram-high-bandwidth-memory-on-knights-landing here] for details.<br />
The nodes are interconnected to the rest of the clusters with QDR Infiniband and shares the regular SciNet GPFS filesystems. <br />
<br />
== Login ==<br />
<br />
First login via ssh with your SciNet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>knl01,knl02,knl03,knl04</tt>'''.<br />
<br />
== KNL Operational Modes ==<br />
<br />
The four nodes all have identical hardware, however there are multiple options that control how the MCDRAM High Bandwidth Memory (HBM) is accessed. Mode changes are not dynamic and require the node to be rebooted to take affect. <br />
<br />
=== Clustering ===<br />
<br />
Currently all KNL nodes have the Cluster Mode configured to "Quadrant". See [http://colfaxresearch.com/knl-numa/ this] article for more details about the clustering options that contorl how memory is accessed on the KNL. <br />
<br />
=== Memory ===<br />
<br />
Two nodes, '''<tt>knl01,kn02</tt>''' have the MCDRAM configured as "Cache" mode and the other two '''<tt>knl03,kn04</tt>''' are configured with the "Flat" memory mode.<br />
See [http://colfaxresearch.com/knl-mcdram/ this ] article for more details about the MCDRAM memory modes. <br />
<br />
Initially when you first compile/port your code, use the Cache mode nodes, however if you wish to try and optimize memory performance by directly using the HBM memory with the [http://colfaxresearch.com/knl-mcdram/#sec-3-2 memkind] library or the [http://colfaxresearch.com/knl-mcdram/#sec-3-1 numactl] options, use the Flat nodes.<br />
<br />
<pre><br />
user@knl03$ numactl --membind 1 ./mycode<br />
</pre><br />
<br />
== Queue ==<br />
<br />
Currently there is no queue, be nice.<br />
<br />
== Software ==<br />
<br />
Software is available using the standard modules framework used on other SciNet systems, however is separate from the GPC modules as the KNL has a newer Centos7 based operating system. <br />
<br />
=== Compilers ===<br />
The Xeon Phi uses the standard intel compilers.<br />
<br />
<pre><br />
module load intel/16.0.3<br />
</pre><br />
<br />
=== MPI ===<br />
<br />
IntelMPI is currently the default MPI<br />
<br />
<pre><br />
module load intelmpi/5.1.3.219<br />
</pre><br />
<br />
== References ==<br />
<br />
* SNUG TechTalk October 2016 - [[Media:Snug_techtalk_KNL.pdf | Intel Xeon Phi Knights Landing ]]<br />
* http://dap.xeonphi.com/#implinks <br />
* http://colfaxresearch.com/get-ready-for-intel-knights-landing-3-papers/<br />
* https://software.intel.com/en-us/articles/getting-ready-for-KNL</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=P8&diff=9378P82018-08-09T17:29:04Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:P8_s822.jpg|center|300px|thumb]]<br />
|name=P8 <br />
|installed=June 2016<br />
|operatingsystem= Linux RHEL 7.2 le / Ubuntu 16.04 le <br />
|loginnode= p8t0[1-2] / p8t0[3-4]<br />
|nnodes= 2x Power8 with 2x NVIDIA K80, 2x Power 8 with 4x NVIDIA P100<br />
|rampernode=512 GB<br />
|corespernode= 2 x 8core (16 physical, 128 SMT)<br />
|interconnect=Infiniband EDR <br />
|vendorcompilers=xlc/xlf, nvcc<br />
}}<br />
<br />
== Specifications==<br />
<br />
The P8 Test System consists of of 4 IBM Power 822LC Servers each with 2x8core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 16 physical cores to support up to 128 threads. 2 nodes have two NVIDIA Tesla K80 GPUs with CUDA Capability 3.7 (Kepler), consisting of 2xGK210 GPUs each with 12 GB of RAM connected using PCI-E, and 2 others have 4x NVIDIA Tesla P100 GPUs each wit h 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.<br />
<br />
== Compile/Devel/Test ==<br />
<br />
First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can ssh to <tt>p8t01</tt> or <tt>p8t02</tt> for the K80 GPUs and to <tt>p8t03</tt> or <tt>p8t04</tt> for the Pascal GPUs.<br />
<br />
== Software for ==<br />
<br />
==== GNU Compilers ====<br />
<br />
To load the newer advance toolchain version use:<br />
<br />
For '''<tt>p8t0[1-2]</tt>''' <br />
<pre><br />
module load gcc/5.3.1<br />
</pre><br />
<br />
For '''<tt>p8t0[3-4]</tt>''' <br />
<pre><br />
module load gcc/6.2.1<br />
</pre><br />
<br />
==== IBM Compilers ====<br />
<br />
To load the native IBM xlc/xlc++ compilers<br />
<br />
For '''<tt>p8t0[1-2]</tt>''' <br />
<pre><br />
module load xlc/13.1.4<br />
module load xlf/13.1.4<br />
</pre><br />
<br />
For '''<tt>p8t0[3-4]</tt>''' <br />
<pre><br />
module load xlc/13.1.5_b2<br />
module load xlf/13.1.5_b2<br />
</pre><br />
<br />
==== Driver Version ====<br />
<br />
The current NVIDIA driver version is 361.93<br />
<br />
==== CUDA ====<br />
<br />
The current installed CUDA Tookit is 8.0<br />
<br />
<pre><br />
module load cuda/8.0<br />
</pre><br />
<br />
The CUDA driver is installed locally, however the CUDA Toolkit is installed in:<br />
<br />
<pre><br />
/usr/local/cuda-8.0<br />
</pre><br />
<br />
==== OpenMPI ====<br />
<br />
Currently OpenMPI has been setup on the four nodes connected over QDR Infiniband.<br />
<br />
For '''<tt>p8t0[1-2]</tt>''' <br />
<pre><br />
$ module load openmpi/1.10.3-gcc-5.3.1<br />
$ module load openmpi/1.10.3-XL-13_15.1.4<br />
</pre><br />
<br />
For '''<tt>p8t0[3-4]</tt>''' <br />
<pre><br />
$ module load openmpi/1.10.3-gcc-6.2.1<br />
$ module load openmpi/1.10.3-XL-13_15.1.5<br />
</pre><br />
<br />
==== PE ====<br />
<br />
IBM's Parallel Environment (PE), is available for use with XL compilers using the following<br />
<br />
<pre><br />
$ module pe/xl.perf<br />
</pre><br />
<br />
<pre><br />
mpiexec -n 4 ./a.out<br />
</pre><br />
<br />
documentation is [http://publib.boulder.ibm.com/epubs/pdf/c2372832.pdf here]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=P7_Linux_Cluster&diff=9377P7 Linux Cluster2018-08-09T17:28:50Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:IBM755.jpg|center|300px|thumb]]<br />
|name=P7 Cluster (P7)<br />
|installed=May 2011, March 2013<br />
|operatingsystem= Linux (RHEL 6.3)<br />
|loginnode= p701 (from <tt>login.scinet</tt>)<br />
|nnodes=8 (256 cores)<br />
|rampernode=128 Gb <br />
|corespernode=32 (128 Threads)<br />
|interconnect=Infiniband (2 DDR/node )<br />
|vendorcompilers=xlc/xlf<br />
|queuetype=LoadLeveler<br />
}}<br />
<br />
== Specifications==<br />
<br />
The P7 Cluster consists of 8 IBM Power 755 Servers each with 4x 8core 3.3GHz Power7 CPUs and 128GB Ram. Similar to the Power 6, the Power 7 utilizes Simultaneous Multi Threading (SMT), but extends the design from 2 threads per core to 4. This allows the 32 physical cores to support up to 128 threads which in many cases can lead to significant speedups.<br />
<br />
== Login ==<br />
<br />
First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can proceed to '''<tt>p7n01-ib0</tt>''' which <br />
is currently the gateway/devel node for this cluster. It is recommended that you modify your .bashrc files to distinguish between the TCS, P7,<br />
and GPC to avoid module confusion, an example configuration is given [[Important_.bashrc_guidelines|here]].<br />
<br />
== Compiler/Devel Node ==<br />
<br />
From '''<tt>p7n01-ib0</tt>''' you can compile, do short tests, and submit your jobs to the queue.<br />
<br />
=== Software ===<br />
==== GNU Compilers ====<br />
gcc/g++/gfortran version 4.4.4 is the default with RHEL 6.3 and is available by default. Gcc 4.6.1 is available as a separate module. However, it is recommended to use the IBM compilers (see below).<br />
<br />
==== IBM Compilers ====<br />
To use the IBM Power specific compilers xlc/xlc++/xlf you need to load the following modules <br />
<pre><br />
$ module load vacpp xlf<br />
</pre><br />
<br />
NOTE: Be sure to use "-q64" when using the IBM compilers.<br />
<br />
==== MPI ====<br />
<br />
IBM's POE is available and will work with both the IBM and GNU compilers.<br />
<pre><br />
$ module load pe<br />
</pre><br />
The mpi wrappers for C, C++ and Fortran 77/90 are mpicc, mpicxx, and mpif77/mpif90, respectively (but mpcc, mpCC and mpfort should also work).<br />
<br />
Note: To use the full C++ bindings of MPI (those in the MPI namespace) in c++ code, you need to add <tt>-cpp</tt> to the compilation command, and you need to add <tt>-Wl,--allow-multiple-definition</tt> to the link command if you are linking several object files that use the MPI c++ bindings.<br />
<br />
<!--<br />
==== OpenMPI ====<br />
<br />
<pre><br />
$ module openmpi/1.5.3-gcc-v4.4.4<br />
</pre><br />
<br />
<pre><br />
$ module openmpi/1.5.3-ibm-11.1+13.1<br />
</pre><br />
--><br />
==== Spark Standalone ====<br />
To run Spark you need to previously load JRE1.7.0 via JDK<br />
<pre><br />
p7n01-$ module load jdk/JRE1.7.0 <br />
</pre><br />
Then load Spark as follow :<br />
<pre><br />
p7n01-$ module load spark/1.4.1<br />
</pre><br />
==== Spark SQL ====<br />
The current build of spark/1.5.0 supports Spark SQL <br />
<pre><br />
p7n01-$ module load jdk/JRE1.7.0 <br />
p7n01-$ module load spark/1.5.0<br />
p7n01-$ module load hadoop/2.3.0<br />
</pre><br />
<br />
== Sample Spark script ==<br />
We recommend you to read the following blog post by Jonathan Dursi to build your first Spark script :<br />
http://www.dursi.ca/spark-in-hpc-clusters/ <br />
<br />
Prior to submitting sparkscript.py, change the import line to<br />
<pre><br />
from pyspark.context import SparkContext<br />
</pre><br />
<br />
Or instead of submitting sparkscript.py, you can also try :<br />
<pre><br />
spark-submit --master $sparkmaster --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/target/spark-examples_2.10-1.4.1.jar 256<br />
</pre><br />
<br />
== Submit a Job ==<br />
<br />
The current Scheduler is IBM's LoadLeveler similar to the implementation on the TCS, however be sure to <br />
include the @environment flags shown below in that sample script as they are different and necessary to get full performance.<br />
<br />
<pre><br />
#!/bin/bash<br />
##===================================<br />
## P7 Load Leveler Submission Script<br />
##===================================<br />
##<br />
## Don't change these parameters unless you really know what you are doing<br />
##<br />
##@ environment = MP_INFOLEVEL=0; MP_USE_BULK_XFER=yes; MP_BULK_MIN_MSG_SIZE=64K; \<br />
## MP_EAGER_LIMIT=64K; MP_DEBUG_ENABLE_AFFINITY=no<br />
##<br />
##===================================<br />
## Avoid core dumps<br />
## @ core_limit = 0<br />
##===================================<br />
## Job specific<br />
##===================================<br />
#<br />
# @ job_name = myjob<br />
# @ job_type = parallel<br />
# @ class = verylong<br />
# @ output = $(jobid).out<br />
# @ error = $(jobid).err<br />
# @ wall_clock_limit = 2:00:00<br />
# @ node = 2<br />
# @ tasks_per_node = 128<br />
# @ queue<br />
#<br />
#===================================<br />
<br />
#./my_script<br />
./my_code <br />
<br />
</pre><br />
<br />
<pre><br />
llsubmit myjob.ll <br />
</pre><br />
<br />
To show running jobs use<br />
<pre><br />
llq<br />
</pre><br />
<br />
To cancel a job use<br />
<pre><br />
llcancel JOBID<br />
</pre><br />
<br />
== Split a Spark job ==<br />
<br />
e.g., To split a job into 256 tasks among 2 workers, you must select 3 nodes (one master and 2 workers) and add the following job specifications :<br />
<pre><br />
#@node = 3<br />
#@preferences = Machine == { "AvailableNode1" "AvailableNode2" "AvailableNode3"}<br />
#@task_per_node = 128<br />
</pre><br />
<br />
== Monitor your (Spark) job from localhost ==<br />
<br />
Spark creates a web UI on each master and slave that you can access from your local web browser. You can notably "check your cluster UI to ensure that workers are registered and have sufficient resources". To do so, you must logged onto P7 (again) with forwarding the port of your cluster UI to your local port (e.g., 9999) :<br />
<pre><br />
ssh -L 9999:masternode:4040 userid@login.scinet.utoronto.ca<br />
</pre><br />
<br />
Then go to your web browser at http://localhost:9999<br />
<br />
<!-- PBS <br />
Currently a vary basic torque (pbs based) queuing system has been setup with one 48 hour batch queue. <br />
<br />
Create a script as follows:<br />
<br />
<pre><br />
#!/bin/bash <br />
# p7script.sh<br />
#PBS -l nodes=1:ppn=128,walltime=1:00:00<br />
#PBS -N P7test<br />
<br />
cd $PBS_O_WORKDIR<br />
<br />
mpirun -np 128 ./a.out<br />
</pre><br />
<br />
Then submit <br />
<pre><br />
qsub p7script.sh<br />
</pre><br />
<br />
Interactive sessions can also be facilitated with<br />
<br />
<pre><br />
qsub -I -l nodes=1:ppn=128,walltime=1:00:00<br />
</pre><br />
<br />
To see running jobs use<br />
<pre><br />
qstat <br />
</pre><br />
<br />
and to cancel a running or queued job<br />
<pre><br />
qdel JOBID<br />
</pre><br />
--></div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=HPSS&diff=9376HPSS2018-08-09T17:28:31Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{|align=right<br />
|align=center|'''Topology Overview'''<br />
|align=center|'''Submission Queue'''<br />
|-<br />
|[[Image:HPSS-overview.png|right|x200px]]<br />
|[[Image:HPSS-queue2.png|right|x200px]]<br />
|-<br />
|<br />
|<br />
|-<br />
|align=center|'''Servers Rack'''<br />
|align=center|'''TS3500 Library'''<br />
|-<br />
|[[Image:HPSS-servers.png|right|x250px]]<br />
|[[Image:HPSS-TS3500.png|right|x250px]]<br />
|}<br />
<br />
= '''High Performance Storage System''' =<br />
<br />
The High Performance Storage System ([http://www.hpss-collaboration.org/index.shtml HPSS] [http://en.wikipedia.org/wiki/High_Performance_Storage_System wikipedia]) is a tape-backed hierarchical storage system that provides a significant portion of the allocated storage space at SciNet. It is a repository for archiving data that is not being actively used. Data can be returned to the active GPFS filesystem when it is needed. <br />
<br />
Since this system is intended for large data storage, it is accessible only to groups who have been awarded storage space at SciNet beyond 5TB in the yearly RAC resource allocation round. However, upon request, any user may be awarded access to HPSS, up to 2TB per group, so that you may get familiar with the system (just email support@scinet.utoronto.ca)<br />
<br />
Access and transfer of data into and out of HPSS is done under the control of the user, whose interaction is expected to be scripted and submitted as a batch job, using one or more of the following utilities:<br />
* [http://www.mgleicher.us/GEL/hsi HSI] is a client with an ftp-like functionality which can be used to archive and retrieve large files. It is also useful for browsing the contents of HPSS.<br />
* [http://www.mgleicher.us/GEL/htar HTAR] is a utility that creates tar formatted archives directly into HPSS. It also creates a separate index file (.idx) that can be accessed and browsed quickly.<br />
* [https://support.scinet.utoronto.ca/wiki/index.php/ISH ISH] is a TUI utility that can perform an inventory of the files and directories in your tarballs.<br />
<br />
We're currently running HPSS v 7.3.3 patch 6, and HSI/HTAR version 4.0.1.2<br />
<br />
== '''Why should I use and trust HPSS?''' ==<br />
* HPSS is a 25 year-old collaboration between IBM and the DoE labs in the US, and is used by about 45 facilities in the [http://www.top500.org “Top 500”] HPC list (plus some black-sites).<br />
* Over 2.5 ExaBytes of combined storage world-wide.<br />
* The top 3 sites in the World report (fall 2017) having 360PB, 220PB and 125PB in production (ECMWF, UKMO and BNL)<br />
* Environment Canada also adopted HPSS in 2017 to store Nav Canada data as well as to serve as their own archive. Currently has 2 X 100PB capacity installed. <br />
* The SciNet HPSS system has been providing nearline capacity for important research data in Canada since early 2011, already at 10PB levels in 2018<br />
* Very reliable, data redundancy and data insurance built-in (dual copies of everything are kept on tapes at SciNet)<br />
* Data on cache and tapes can be geo-distributed for further resilience and HA.<br />
* Highly scalable; current performance at SciNet - after a modest upgrade in 2017 - Ingest: ~150 TB/day, Recall: ~45 TB/day (aggregated).<br />
* HSI/HTAR clients also very reliable and used on several HPSS sites. ISH was written at SciNet.<br />
* [[Media:HPSS_rationale_SNUG.pdf|HPSS fits well with the Storage Capacity Expansion Plan at SciNet]] (pdf presentation)<br />
<br />
== '''Guidelines''' ==<br />
* Expanded storage capacity is provided on tape -- a media that is not suited for storing small files. Files smaller than ~200MB should be grouped into tarballs with tar or htar.<br />
* Optimal performance for aggregated transfers and allocation on tapes is obtained with [[Why not tarballs too large |<font color=red>tarballs of size 500GB or less</font>]], whether ingested by htar or hsi ([[Why not tarballs too large | <font color=red>WHY?</font>]])<br />
* We strongly urge that you use the sample scripts we are providing as the basis for your job submissions.<br />
* Make sure to check the application's exit code and returned logs for errors after any data transfer or tarball creation process<br />
<br />
== '''New to the System?''' ==<br />
The first step is to email scinet support and request an HPSS account (or else you will get "Error - authentication/initialization failed" and 71 exit codes). <br />
<br />
THIS set of instructions on the wiki is the best and most compressed "manual" we have. It may seem a bit overwhelming at first, because of all the job script templates we make available below (they are here so you don't have to think <br />
too much, just copy and paste), but if you approach the index at the top as a "case switch" mechanism for what you intend to do, everything falls in place.<br />
<br />
Try this sequence:<br />
<br />
1) [https://wiki.scinet.utoronto.ca/wiki/index.php/HPSS#Access_Through_an_Interactive_HSI_session take a look around HPSS using an interactive HSI session]<br />
<br />
(most linux shell commands have an equivalent in HPSS)<br />
<br />
2) [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_tarball_create archive a small test directory using HTAR]<br />
<br />
2a) use step 1) to see what happened<br />
<br />
3) [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_data_offload archive a file using hsi]<br />
<br />
3a) use step 1) to see what happened<br />
<br />
4) [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_transferring_directories archive a small test directory using HSI]<br />
<br />
4a) use step 1) to see what happened<br />
<br />
5) now try the other cases and so on. In a couple of hours you'll be in pretty good shape.<br />
<br />
== '''Bridge between BGQ and HPSS''' ==<br />
<br />
At this time BGQ users will have to migrate data to Niagara scratch prior to transferring it to HPSS. We are looking for ways to improve this workflow.<br />
<br />
== '''Access Through the Queue System''' ==<br />
All access to the archive system is done through the [https://docs.computecanada.ca/wiki/Niagara_Quickstart#Submitting_jobs NIA queue system].<br />
<br />
* Job submissions should be done to the 'archivelong' queue or the 'archiveshort'<br />
* Short jobs are limited to 1H walltime by default. Long jobs (> 1H) are limited to 72H walltime.<br />
* Users are limited to only 2 long jobs and 2 short jobs at the same time, and 10 jobs total on the each queue.<br />
* There can only be 5 long jobs running at any given time overall. Remaining submissions will be placed on hold for the time being. So far we have not seen a need for overall limit on short jobs.<br />
<br />
The status of pending jobs can be monitored with squeue specifying the archive queue:<br />
<pre><br />
squeue -p archiveshort<br />
OR<br />
squeue -p archivelong<br />
</pre><br />
<br />
== '''Access Through an Interactive HSI session''' ==<br />
* You may want to acquire an interactive shell, start an HSI session and navigate the archive naming-space. Keep in mind, you're restricted to 1H.<br />
<br />
<pre><br />
pinto@nia-login07:~$ salloc -p archiveshort -t 1:00:00<br />
salloc: Granted job allocation 50918<br />
salloc: Waiting for resource configuration<br />
salloc: Nodes hpss-archive02-ib are ready for job<br />
hpss-archive02-ib:~$<br />
<br />
hpss-archive02-ib:~$ hsi (DON'T FORGET TO START HSI)<br />
******************************************************************<br />
* Welcome to HPSS@SciNet - High Perfomance Storage System *<br />
* * <br />
* INFO: THIS IS THE NEW 7.5.1 HPSS SYSTEM! *<br />
* *<br />
* Contact Information: support@scinet.utoronto.ca *<br />
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *<br />
* CHECK THE INTEGRITY OF YOUR TARBALLS *<br />
****************************************************************** <br />
[HSI]/archive/s/scinet/pinto-> ls<br />
<br />
[HSI]/archive/s/scinet/pinto-> cd <some directory><br />
</pre><br />
<br />
=== Scripted File Transfers ===<br />
File transfers in and out of the HPSS should be scripted into jobs and submitted to the ''archivelong'' queue or the ''archiveshort'' . See generic example below:<br />
<br />
<source lang="bash"><br />
#!/bin/bash -l<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J htar_create_tarball_in_hpss<br />
#SBATCH --mail-type=ALL<br />
<br />
echo "Creating a htar of finished-job1/ directory tree into HPSS"<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be $ARCHIVE<br />
<br />
DEST=$ARCHIVE/finished-job1.tar<br />
<br />
# htar WILL overwrite an existing file with the same name so check beforehand.<br />
<br />
hsi ls $DEST &> /dev/null<br />
status=$?<br />
<br />
if [ $status == 0 ]; then <br />
echo 'File $DEST already exists. Nothing has been done'<br />
exit 1<br />
fi<br />
<br />
cd $SCRATCH/workarea/ <br />
htar -Humask=0137 -cpf $ARCHIVE/finished-job1.tar finished-job1/ <br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HTAR returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
'''Note:''' Always trap the execution of your jobs for abnormal terminations, and be sure to return the exit code<br />
<br />
=== Job Dependencies ===<br />
<br />
Typically data will be recalled to /scratch when it is needed for analysis. Job dependencies can be constructed so that analysis jobs wait in the queue for data recalls before starting. The qsub flag is<br />
<pre><br />
--dependency=<type:JOBID><br />
</pre><br />
where JOBID is the job number of the archive recalling job that must finish successfully before the analysis job can start.<br />
<br />
Here is a short cut for generating the dependency (lookup [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_data_recall data-recall.sh samples]):<br />
<pre><br />
hpss-archive02-ib:~$ sbatch $(sbatch data-recall.sh | awk {print "--dependency=afterany:"$1}') job-to-work-on-recalled-data.sh<br />
</pre><br />
<br />
== '''HTAR''' ==<br />
''' Please aggregate small files (<~200MB) into tarballs or htar files. '''<br />
<br />
''' [[Why not tarballs too large |<font color=red>Keep your tarballs to size 500GB or less</font>]], whether ingested by htar or hsi ([[Why not tarballs too large | <font color=red>WHY?</font>]])'''<br />
<br />
HTAR is a utility that is used for aggregating a set of files and directories, by using a sophisticated multithreaded buffering scheme to write files directly from GPFS into HPSS, creating an archive file that conforms to the POSIX TAR specification, thereby achieving a high rate of performance. HTAR does not do gzip compression, however it already has a built-in checksum algorithm.<br />
<br />
'''Caution'''<br />
* Files larger than 68 GB cannot be stored in an HTAR archive. If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI.<br />
* Files with pathnames too long will be skipped (greater than 100 characters), so as to conform with TAR protocol [[(POSIX 1003.1 USTAR)]] -- Note that the HTAR will erroneously indicate success, however will produce exit code 70. For now, you can check for this type of error by "grep Warning my.output" after the job has completed.<br />
* Unlike with cput/cget in HSI, "prompt before overwrite", this is not the default with (h)tar. Be careful not to unintentionally overwrite a previous htar destination file in HPSS. There could be a similar situation when extracting material back into GPFS and overwriting the originals. Be sure to double-check the logic in your scripts.<br />
* Check the HTAR exit code and log file before removing any files from the GPFS active filesystems.<br />
<br />
<br />
=== HTAR Usage ===<br />
* To write the ''file1'' and ''file2'' files to a new archive called ''files.tar'' in the default HPSS home directory, and preserve mask attributes (-p), enter:<br />
<pre><br />
htar -cpf files.tar file1 file2<br />
OR<br />
htar -cpf $ARCHIVE/files.tar file1 file2<br />
</pre><br />
<br />
* To write a ''subdirA'' to a new archive called ''subdirA.tar'' in the default HPSS home directory, enter:<br />
<pre><br />
htar -cpf subdirA.tar subdirA<br />
</pre><br />
<br />
* To extract all files from the archive file called ''proj1.tar'' in HPSS into the ''project1/src'' directory in GPFS, and use the time of extraction as the modification time, enter:<br />
<pre><br />
cd project1/src<br />
htar -xpmf proj1.tar<br />
</pre><br />
<br />
* To display the names of the files in the ''out.tar'' archive file within the HPSS home directory, enter (the out.tar.idx file will be queried):<br />
<pre><br />
htar -vtf out.tar<br />
</pre><br />
<br />
* To ensure that both the htar and the .idx files have read permissions to other members in your group use the umask option<br />
<pre><br />
htar -Humask=0137 ....<br />
</pre><br />
<br />
For more details please check the '''[http://www.mgleicher.us/GEL/htar/ HTAR - Introduction]''' or the '''[http://www.mgleicher.us/GEL/htar/htar_man_page.html HTAR Man Page]''' online<br />
<br />
<br />
==== Sample tarball create ====<br />
<source lang="bash"><br />
#!/bin/bash -l<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J htar_create_tarball_in_hpss<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be /archive/$(id -gn)/$(whoami)/<br />
<br />
DEST=$ARCHIVE/finished-job1.tar<br />
<br />
# htar WILL overwrite an existing file with the same name so check beforehand.<br />
<br />
hsi ls $DEST &> /dev/null<br />
status=$?<br />
<br />
if [ $status == 0 ]; then <br />
echo 'File $DEST already exists. Nothing has been done'<br />
exit 1<br />
fi<br />
<br />
cd $SCRATCH/workarea/ <br />
htar -Humask=0137 -cpf $DEST finished-job1/ <br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HTAR returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
'''Note:''' If you attempt to start a transfer with any files larger than 68GB the whole HTAR session will fail, and you'll get a notification listing all those files, so that you can transfer them with HSI. <br />
<pre><br />
----------------------------------------<br />
INFO: File too large for htar to handle: finished-job1/file1 (86567185745 bytes)<br />
INFO: File too large for htar to handle: finished-job1/file2 (71857244579 bytes)<br />
ERROR: 2 oversize member files found - please correct and retry<br />
ERROR: [FATAL] error(s) generating filename list <br />
HTAR: HTAR FAILED<br />
###WARNING htar returned non-zero exit status<br />
</pre><br />
<br />
==== Sample tarball list ====<br />
<source lang="bash"><br />
#!/bin/bash -l<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J htar_list_tarball_in_hpss<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be $ARCHIVE<br />
<br />
DEST=$ARCHIVE/finished-job1.tar<br />
<br />
htar -tvf $DEST<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HTAR returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
==== Sample tarball extract ====<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J htar_extract_tarball_from_hpss<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be $ARCHIVE<br />
<br />
cd $SCRATCH/recalled-from-hpss<br />
htar -xpmf $ARCHIVE/finished-job1.tar<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HTAR returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
== '''HSI''' ==<br />
<br />
HSI may be the primary client with which some users will interact with HPSS. It provides an ftp-like interface for archiving and retrieving tarballs or [https://support.scinet.utoronto.ca/wiki/index.php/HPSS#Sample_transferring_directories directory trees]. In addition it provides a number of shell-like commands that are useful for examining and manipulating the contents in HPSS. The most commonly used commands will be:<br />
{|border="1" cellpadding="10" cellspacing="0"<br />
|-<br />
| cput <br />
| Conditionally saves or replaces a HPSSpath file to GPFSpath if the GPFS version is new or has been updated<br />
cput [options] GPFSpath [: HPSSpath]<br />
|-<br />
| cget <br />
| Conditionally retrieves a copy of a file from HPSS to GPFS only if a GPFS version does not already exist. <br />
cget [options] [GPFSpath :] HPSSpath<br />
|-<br />
| cd,mkdir,ls,rm,mv<br />
| Operate as one would expect on the contents of HPSS.<br />
|-<br />
| lcd,lls<br />
| ''Local'' commands to GPFS<br />
|}<br />
<br />
<br />
*There are 3 distinctions about HSI that you should keep in mind, and that can generate a bit of confusion when you're first learning how to use it:<br />
** HSI doesn't currently support renaming directories paths during transfers on-the-fly, therefore the syntax for cput/cget may not work as one would expect in some scenarios, requiring some workarounds.<br />
** HSI has an operator ":" which separates the GPFSpath and HPSSpath, and must be surrounded by whitespace (one or more space characters)<br />
** The order for referring to files in HSI syntax is different from FTP. In HSI the general format is always the same, GPFS first, HPSS second, cput or cget:<br />
<pre><br />
GPFSfile : HPSSfile<br />
</pre><br />
For example, when using HSI to store the tarball file from GPFS into HPSS, then recall it to GPFS, the following commands could be used:<br />
<pre><br />
cput tarball-in-GPFS : tarball-in-HPSS<br />
cget tarball-recalled : tarball-in-HPSS<br />
</pre><br />
<br />
unlike with FTP, where the following syntax would be used:<br />
<pre><br />
put tarball-in-GPFS tarball-in-HPSS <br />
get tarball-in-HPSS tarball-recalled<br />
</pre><br />
* Simple commands can be executed on a single line.<br />
<pre><br />
hsi "mkdir LargeFilesDir; cd LargeFilesDir; cput tarball-in-GPFS : tarball-in-HPSS"<br />
</pre><br />
<br />
* More complex sequences can be performed using an except such as this:<br />
<pre><br />
hsi <<EOF<br />
mkdir LargeFilesDir<br />
cd LargeFilesDir<br />
cput tarball-in-GPFS : tarball-in-HPSS<br />
lcd $SCRATCH/LargeFilesDir2/<br />
cput -Ruph * <br />
end<br />
EOF<br />
</pre><br />
<br />
* The commands below are equivalent, but we recommend that you always use full path, and organize the contents of HPSS, where the default HSI directory placement is $ARCHIVE:<br />
<pre><br />
hsi cput tarball<br />
hsi cput tarball : tarball<br />
hsi cput $SCRATCH/tarball : $ARCHIVE/tarball<br />
</pre><br />
<br />
* There are no known issues renaming files on-the-fly:<br />
<pre><br />
hsi cput $SCRATCH/tarball1 : $ARCHIVE/tarball2<br />
hsi cget $SCRATCH/tarball3 : $ARCHIVE/tarball2<br />
</pre><br />
<br />
* However the syntax forms such as the ones below will fail, since they rename the directory paths.<br />
<pre><br />
hsi cput -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)<br />
OR<br />
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir2 (FAILS)<br />
OR<br />
hsi cput -Ruph $SCRATCH/LargeFilesDir/* : $ARCHIVE/LargeFilesDir2 (FAILS)<br />
OR<br />
hsi cget -Ruph $SCRATCH/LargeFilesDir : $ARCHIVE/LargeFilesDir (FAILS)<br />
</pre><br />
<br />
One workaround is the following 2-steps process, where you do a "lcd " in GPFS first, and recursively transfer the whole directory (-R), keeping the same name. You may use '-u' option to resume a previously disrupted session, and the '-p' to preserve timestamp, and '-h' to keep the links.<br />
<pre><br />
hsi <<EOF<br />
lcd $SCRATCH<br />
cget -Ruph LargeFilesDir<br />
end<br />
EOF<br />
</pre><br />
<br />
Another workaround is do a "lcd" into the GPFSpath first and a "cd" in the HPSSpath, but transfer the files individually with the '*' wild character. This option lets you change the directory name:<br />
<pre><br />
hsi <<EOF<br />
lcd $SCRATCH/LargeFilesDir<br />
mkdir $ARCHIVE/LargeFilesDir2<br />
cd $ARCHIVE/LargeFilesDir2<br />
cput -Ruph * <br />
end<br />
EOF<br />
</pre><br />
<br />
=== Documentation === <br />
Complete documentation on HSI is available from the Gleicher Enterprises links below. You may peruse those links and come with alternative syntax forms. You may even be already familiar with HPSS/HSI from other HPC facilities, that may or not have procedures similar to ours. HSI doesn't always work as expected when you go outside of our recommended syntax, so '''we strongly urge that you use the sample scripts we are providing as the basis''' for your job submissions<br />
* [http://www.mgleicher.us/hsi/hsi_reference_manual_2/introduction.html HSI Introduction]<br />
* [http://www.mgleicher.us/hsi/hsi_man_page.html man hsi]<br />
* [http://support.scinet.utoronto.ca/wiki/index.php/HSI_help hsi help]<br />
* [http://www.mgleicher.us/hsi/hsi-exit-codes.html exit codes] <br />
'''Note:''' HSI returns the highest-numbered exit code, in case of multiple operations in the same hsi session. You may use '/scinet/niagara/bin/exit2msg $status' to translate those codes into intelligible messages<br />
<br />
=== Typical Usage Scripts===<br />
The most common interactions will be ''putting'' data into HPSS, examining the contents (ls,ish), and ''getting'' data back onto GPFS for inspection or analysis.<br />
<br />
==== Sample '''data offload''' ====<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-offload.sh<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong <br />
#SBATCH -N 1<br />
#SBATCH -J offload<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# individual tarballs already exist<br />
<br />
/usr/local/bin/hsi -v <<EOF1<br />
mkdir put-away<br />
cd put-away<br />
cput $SCRATCH/workarea/finished-job1.tar.gz : finished-job1.tar.gz<br />
end<br />
EOF1<br />
status=$?<br />
if [ ! $status == 0 ];then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
/usr/local/bin/hsi -v <<EOF2<br />
mkdir put-away<br />
cd put-away<br />
cput $SCRATCH/workarea/finished-job2.tar.gz : finished-job2.tar.gz<br />
end<br />
EOF2<br />
status=$?<br />
if [ ! $status == 0 ];then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
trap - TERM INT<br />
</source><br />
<br />
<br />
'''Note:''' as in the above example, we recommend that you capture the (highest-numbered) exit code for each hsi session independently. And remember, you may improve your exit code verbosity by adding the excerpt below to your scripts:<br />
<source lang="bash"><br />
if [ ! $status == 0 ];then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
==== Sample '''data list''' ====<br />
A very trivial way to list the contents of HPSS would be to just submit the HSI 'ls' command.<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-list.sh<br />
#SBATCH -t 1:00:00<br />
#SBATCH -p archiveshort<br />
#SBATCH -N 1<br />
#SBATCH -J hpss_ls<br />
#SBATCH --mail-type=ALL<br />
<br />
/usr/local/bin/hsi -v <<EOF<br />
cd put-away<br />
ls -R<br />
end<br />
EOF<br />
</source><br />
''Warning: if you have a lot of files, the ls command will take a long time to complete. For instance, about 400,000 files can be listed in about an hour. Adjust the walltime accordingly, and be on the safe side.''<br />
<br />
However, we provide a much more useful and convenient way to explore the contents of HPSS with the inventory shell [[ISH]]. This example creates an index of all the files in a user's portion of the namespace. The list is placed in the directory /home/$(whoami)/.ish_register that can be inspected from the login nodes.<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-list.sh<br />
#SBATCH -t 1:00:00<br />
#SBATCH -p archiveshort<br />
#SBATCH -N 1<br />
#SBATCH -J hpss_index<br />
#SBATCH --mail-type=ALL<br />
<br />
INDEX_DIR=$HOME/.ish_register<br />
if ! [ -e "$INDEX_DIR" ]; then<br />
mkdir -p $INDEX_DIR<br />
fi<br />
<br />
export ISHREGISTER="$INDEX_DIR"<br />
/scinet/niagara/bin/ish hindex<br />
</source><br />
''Note: the above warning on collecting the listing for many files applies here too.''<br />
<br />
This index can be browsed or searched with ISH on the development nodes.<br />
<source lang="bash"><br />
hpss-archive02-ib:~$ /scinet/niagara/bin/ish ~/.ish_register/hpss.igz <br />
[ish]hpss.igz> help<br />
</source><br />
<br />
ISH is a powerful tool that is also useful for creating and browsing indices of tar and htar archives, so please look at the [[ISH|documentation]] or built in help.<br />
<br />
==== Sample '''data recall''' ====<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-recall.sh<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J recall_files<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
<br />
mkdir -p $SCRATCH/recalled-from-hpss<br />
<br />
# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder<br />
hsi -v << EOF<br />
cget $SCRATCH/recalled-from-hpss/Jan-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Jan-2010-jobs.tar.gz<br />
cget $SCRATCH/recalled-from-hpss/Feb-2010-jobs.tar.gz : $ARCHIVE/put-away-on-2010/Feb-2010-jobs.tar.gz<br />
end<br />
EOF<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagar/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
<br />
We should emphasize that a single ''cget'' of multiple files (rather than several separate gets) allows HSI to do optimization, as in the following example:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-recall.sh<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J recall_files_optimized<br />
#SBATCH --mail-type=AL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
mkdir -p $SCRATCH/recalled-from-hpss<br />
<br />
# individual tarballs previously organized in HPSS inside the put-away-on-2010/ folder<br />
hsi -v << EOF<br />
lcd $SCRATCH/recalled-from-hpss/<br />
cd $ARCHIVE/put-away-on-2010/<br />
cget Jan-2010-jobs.tar.gz Feb-2010-jobs.tar.gz<br />
end<br />
EOF<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
==== Sample '''transferring directories''' ====<br />
Remember, it's not possible to rename directories on-the-fly:<br />
<pre><br />
hsi cget -Ruph $SCRATCH/LargeFiles-recalled : $ARCHIVE/LargeFiles (FAILS)<br />
</pre><br />
<br />
One workaround is transfer the whole directory (and sub-directories) recursively:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-recall.sh<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J recall_directories<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
<br />
mkdir -p $SCRATCH/recalled<br />
<br />
hsi -v << EOF<br />
lcd $SCRATCH/recalled<br />
cd $ARCHIVE/<br />
cget -Ruph LargeFiles<br />
end<br />
EOF<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
<br />
Another workaround is to transfer files and subdirectories individually with the "*" wild character:<br />
<source lang="bash"><br />
#!/bin/bash<br />
# This script is named: data-recall.sh<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J recall_directories<br />
#SBATCH --mail-type=AL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
<br />
mkdir -p $SCRATCH/LargeFiles-recalled<br />
<br />
hsi -v << EOF<br />
lcd $SCRATCH/LargeFiles-recalled<br />
cd $ARCHIVE/LargeFiles<br />
cget -Ruph *<br />
end<br />
EOF<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
<br />
<br />
* For more details please check the '''[http://www.mgleicher.us/GEL/hsi/ HSI Introduction]''', the '''[http://www.mgleicher.us/GEL/hsi/hsi_man_page.html HSI Man Page]''' or the or the [https://support.scinet.utoronto.ca/wiki/index.php/HSI_help '''hsi help''']<br />
<br />
== '''[[ISH|ISH]]''' ==<br />
=== [[ISH|Documentation and Usage]] ===<br />
<br />
== '''File and directory management''' ==<br />
=== Moving/renaming ===<br />
* you may use 'mv' or 'cp' in the same way as the linux version.<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J deletion_script<br />
#SBATCH --mail-type=ALL<br />
<br />
echo "HPSS file and directory management"<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
<br />
/usr/local/bin/hsi -v <<EOF1<br />
mkdir $ARCHIVE/2011<br />
mv $ARCHIVE/oldjobs $ARCHIVE/2011<br />
cp -r $ARCHIVE/almostfinished/*done $ARCHIVE/2011<br />
end<br />
EOF1<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
=== Deletions ===<br />
==== Recommendations ====<br />
* Be careful with the use of 'cd' commands to non-existing directories before the 'rm' command. Results may be unpredictable<br />
* Avoid the use of the stand alone wild character '''*'''. If necessary, whenever possible have it bound to common patterns, such as '*.tmp', so to limit unintentional mis-happens<br />
* Avoid using relative paths, even the env variable $ARCHIVE. Better to explicitly expand the full paths in your scripts<br />
* Avoid using recursive/looped deletion instructions on $SCRATCH contents from the archive job scripts. Even on $ARCHIVE contents, it may be better to do it as an independent job submission, after you have verified that the original ingestion into HPSS finished without any issues.<br />
<br />
==== Typical example ====<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J deletion_script<br />
#SBATCH --mail-type=ALL<br />
<br />
echo "Deletion of an outdated directory tree into HPSS"<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that the initial directory in HPSS ($ARCHIVE) has the path explicitly expanded<br />
<br />
/usr/local/bin/hsi -v <<EOF1<br />
rm /archive/s/scinet/pinto/*.tmp<br />
rm -R /archive/s/scinet/pinto/obsolete<br />
end<br />
EOF1<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
==== Deleting with an interactive HSI session ====<br />
* You may feel more comfortable acquiring an interactive shell, starting an HSI session and proceeding with your deletions that way. Keep in mind, you're restricted to 1H.<br />
<br />
* After using the ''qsub -q archive -I'' command you'll get a standard shell prompt on an archive execution node (hpss-archive02), as you would on any compute node. However you will need to run '''HSI''' or '''HTAR''' to access resources on HPSS. <br />
<br />
* HSI will give you a prompt very similar to a standard shell, where your can navigate around using commands such 'ls', 'cd', 'pwd', etc ... NOTE: not every bash command has an equivalent on HSI - for instance, you can not 'vi' or 'cat'.<br />
<br />
<pre><br />
pinto@nia-login07:~$ salloc -p archiveshort -t 1:00:00<br />
salloc: Granted job allocation 50359<br />
salloc: Waiting for resource configuration<br />
salloc: Nodes hpss-archive02-ib are ready for job<br />
<br />
hpss-archive02-ib:~$ hsi<br />
******************************************************************<br />
* Welcome to HPSS@SciNet - High Perfomance Storage System *<br />
* * <br />
* INFO: THIS IS THE NEW 7.5.1 HPSS SYSTEM! *<br />
* *<br />
* Contact Information: support@scinet.utoronto.ca *<br />
* NOTE: do not transfer SMALL FILES with HSI. Use HTAR instead *<br />
* CHECK THE INTEGRITY OF YOUR TARBALLS *<br />
******************************************************************<br />
<br />
[HSI]/archive/s/scinet/pinto-> rm -R junk<br />
<br />
</pre><br />
<br />
== '''HPSS for the 'Watchmaker' ''' ==<br />
=== Efficient alternative to htar ===<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J tar_create_tarball_in_hpss_with_hsi_by_piping<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be $ARCHIVE<br />
<br />
# When using a pipeline like this<br />
set -o pipefail <br />
<br />
# to put (cput will fail)<br />
tar -c $SCRATCH/mydir | hsi put - : $ARCHIVE/mydir.tar<br />
status=$?<br />
if [ ! $status == 0 ]; then<br />
echo 'TAR+HSI+piping returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
# to immediately generate an index<br />
ish hindex $ARCHIVE/mydir.tar<br />
status=$?<br />
if [ ! $status == 0 ]; then<br />
echo 'ISH returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
# to get<br />
#cd $SCRATCH<br />
#hsi cget - : $ARCHIVE/mydir.tar | tar -xv <br />
#status=$?<br />
# if [ ! $status == 0 ]; then<br />
# echo 'TAR+HSI+piping returned non-zero code.'<br />
# /scinet/niagara/bin/exit2msg $status<br />
# exit $status<br />
#else<br />
# echo 'TRANSFER SUCCESSFUL'<br />
#fi<br />
<br />
trap - TERM INT<br />
<br />
</source><br />
'''Notes:''' <br />
* Combining commands in this fashion, besides being HPSS-friendly, should not be that noticeably slower than the recursive put with HSI that stores each file one by one. However, reading the files back from tape in this format will be many times faster. It would also overcome the current 68GB limit on the size of stored files that we have with htar.<br />
* To top things off, we recommend indexing with ish (in the same script) immediately after the tarball creation , while it resides in the HPSS cache. It would be as if htar was used.<br />
* To ensure that an error at any stage of the pipeline shows up in the returned status use: ''set -o pipefail'' (The default is to return the status of the last command in the pipeline and this is not what you want.)<br />
* Optimal performance for aggregated transfers and allocation on tapes is obtained with [[Why not tarballs too large |<font color=red>tarballs of size 500GB or less</font>]], whether ingested by htar or hsi ([[Why not tarballs too large | <font color=red>WHY?</font>]]). Be sure to check the contents of the directory tree with 'du' for the total amount of data before sending them to the tar+HSI piping.<br />
<br />
=== Multi-threaded gzip'ed compression with pigz ===<br />
We compiled multi-threaded implementation of gzip called pigz (http://zlib.net/pigz/). It's now part of the "extras" module. It can also be used on any compute or devel nodes. This makes the execution of the previous version of the script much quicker than if you were to use 'tar -cfz'. In addition, by piggy-backing ISH to the end of the script, it will know what to do with the just created mydir.tar.gz compressed tarball.<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J tar_create_compressed_tarball_in_hpss_with_hsi_by_piping<br />
#SBATCH --mail-type=ALL<br />
<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be $ARCHIVE<br />
<br />
# When using a pipeline like this<br />
set -o pipefail <br />
<br />
load module extras<br />
<br />
# to put (cput will fail)<br />
tar -c $SCRATCH/mydir | pigz | hsi put - : $ARCHIVE/mydir.tar.gz<br />
status=$?<br />
if [ ! $status == 0 ]; then<br />
echo 'TAR+PIGZ+HSI+piping returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
=== Content Verification ===<br />
<br />
==== HTAR CRC checksums ====<br />
Specifies that HTAR should generate CRC checksums when creating the archive.<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J htar_create_tarball_in_hpss_with_checksum_verification<br />
#SBATCH --mail-type=ALL<br />
<br />
trap "echo 'Job script not completed';exit 129" TERM INT<br />
# Note that your initial directory in HPSS will be $ARCHIVE<br />
<br />
cd $SCRATCH/workarea<br />
<br />
# to put<br />
htar -Humask=0137 -cpf $ARCHIVE/finished-job1.tar -Hcrc -Hverify=1 finished-job1/<br />
<br />
# to get<br />
#mkdir $SCRATCH/verification<br />
#cd $SCRATCH/verification<br />
#htar -Hcrc -xvpmf $ARCHIVE/finished-job1.tar <br />
<br />
status=$?<br />
<br />
trap - TERM INT<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HTAR returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
==== Current HSI version - Checksum built-in ====<br />
<br />
MD5 is the standard Hashing Algorithm for the HSI build at SciNet. For hsi ingestions with the '-c on' option you should be able to query the md5 hash with the hsi command 'hashli'. That value is stored as an UDA (User Defined Attribute) for each file (a feature of HPSS starting with 7.4)<br />
<br />
[http://www.mgleicher.us/GEL/hsi/hsi/hsi_reference_manual_2/checksum-feature.html More usage details here]<br />
<br />
The checksum algorithm is very CPU-intensive. Although the checksum code is compiled with a high level of compiler optimization, transfer rates can be significantly reduced when checksum creation or verification is in effect. The amount of degradation in transfer rates depends on several factors, such as processor speed, network transfer speed, and speed of the local filesystem (GPFS).<br />
<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J MD5_checksum_verified_transfer<br />
#SBATCH --mail-type=ALL<br />
<br />
thefile=<GPFSpath><br />
storedfile=<HPSSpath><br />
<br />
# Generate checksum on fly (-c on)<br />
hsi -q put -c on $thefile : $storedfile<br />
pid=$!<br />
<br />
# Check the exit code of the HSI process <br />
status=$?<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
# verify checksum<br />
hsi lshash $storedfile<br />
status=$?<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
# get the file back with checksum<br />
hsi get -c on $storedfile<br />
status=$?<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
==== Prior to HSI version 4.0.1.1 ====<br />
<br />
This will checksum the contents of the HPSSpath against the original GPFSpath after the transfer has finished.<br />
<source lang="bash"><br />
#!/bin/bash<br />
#SBATCH -t 72:00:00<br />
#SBATCH -p archivelong<br />
#SBATCH -N 1<br />
#SBATCH -J checksum_verified_transfer<br />
#SBATCH --mail-type=ALL<br />
<br />
thefile=<GPFSpath><br />
storedfile=<HPSSpath><br />
<br />
# Generate checksum on fly using a named pipe so that file is only read from GPFS once<br />
mkfifo /tmp/NPIPE<br />
cat $thefile | tee /tmp/NPIPE | hsi -q put - : $storedfile &<br />
pid=$!<br />
md5sum /tmp/NPIPE |tee /tmp/$fname.md5<br />
rm -f /tmp/NPIPE<br />
<br />
# Check the exit code of the HSI process <br />
wait $pid<br />
status=$?<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
<br />
# change filename to stdin in checksum file<br />
sed -i.1 "s+/tmp/NPIPE+-+" /tmp/$fname.md5<br />
<br />
# verify checksum<br />
hsi -q get - : $storedfile | md5sum -c /tmp/$fname.md5<br />
status=$?<br />
<br />
if [ ! $status == 0 ]; then<br />
echo 'HSI returned non-zero code.'<br />
/scinet/niagara/bin/exit2msg $status<br />
exit $status<br />
else<br />
echo 'TRANSFER SUCCESSFUL'<br />
fi<br />
</source><br />
<br />
== '''Access to HPSS using Globus''' ==<br />
* <font color=red> Please note that Globus access to HPSS is disabled until further notice, due to lack of version compatibility.</font><br />
<br />
* You may now transfer data between SciNet's HPSS and an external source<br />
* Follow the link below <br />
https://globus.computecanada.ca<br />
: Enter your Compute Canada username and password.<br />
* In the 'File Transfer' tab, enter ''''Compute Canada HPSS'''' as one of the Endpoints. To authenticate this endpoint, enter your SciNet username and password.<br />
* You may read more about Compute Canada's Globus Portal here:<br />
https://docs.computecanada.ca/wiki/Globus<br />
<br />
== '''Access to HPSS using SME''' ==<br />
* Storage Made Easy - SME - is an Enterprise Cloud Portal adopted by SciNet to allow our users to access HPSS<br />
* Best suitable for light transfers to/from your personal computer and to navigate your contents on HPSS<br />
* Follow the link below using a web browser and login with your SicNet UserID and password. Under File Manager you will find the "'''SciNet HPSS'''" folder.<br />
https://sme.scinet.utoronto.ca<br />
* SME can be configured as a DropBox. To download the Free Cloud File Manager native to your OS (Windows, Mac, Linux, mobile), follow the link below:<br />
https://www.storagemadeeasy.com/clients_and_tools/<br />
Once you have downloaded and installed the Cloud Manager App, fill up the following information:<br />
Server location<br />
https://sme.scinet.utoronto.ca/api<br />
* You may learn more about SME capabilities and features here:<br />
https://www.storagemadeeasy.com/ownFileserver/<br />
https://www.storagemadeeasy.com/pricing/#features (Enterprise)<br />
https://storagemadeeasy.com/faq/<br />
<br />
== '''User provided Content/Suggestions''' ==<br />
== '''[[HPSS-by-pomes|Packing up large data sets and putting them on HPSS]]''' ==<br />
(Pomés group recommendations)<br />
<br />
[[Data Management|BACK TO Data Management]]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=SOSCIP_GPU&diff=9375SOSCIP GPU2018-08-09T17:28:05Z<p>Rzon: </p>
<hr />
<div>__NOTOC__<br />
<br />
{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:S882lc.png|center|300px|thumb]]<br />
|name=SOSCIP GPU <br />
|installed=September 2017<br />
|operatingsystem= Ubuntu 16.04 le <br />
|loginnode= sgc01 <br />
|nnodes= 14x Power 8 with 4x NVIDIA P100<br />
|rampernode=512 GB<br />
|corespernode= 2 x 10core (20 physical, 160 SMT)<br />
|interconnect=Infiniband EDR <br />
|vendorcompilers=xlc/xlf, nvcc<br />
}}<br />
<br />
== SOSCIP ==<br />
<br />
The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].<br />
<br />
== Support Email ==<br />
<br />
Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.<br />
<br />
== Specifications==<br />
<br />
The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.<br />
<br />
== Access and Login ==<br />
<br />
In order to obtain access to the system, you must request access to the SOSCIP GPU Platform. Instructions will have been sent to your sponsoring faculty member via E-mail at the beginning of your SOSCIP project.<br />
<br />
Access to the SOSCIP GPU Platform is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>''' via ssh. Your user name and password is the same as it is for SciNet systems.<br />
<br />
== Filesystem ==<br />
<br />
The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.<br />
<br />
== Job Submission ==<br />
<br />
The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.<br />
<br />
<pre><br />
$ sbatch myjob.script<br />
</pre><br />
<br />
Where myjob.script is <br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
hostname<br />
nvidia-smi<br />
</pre><br />
<br />
More information about the <tt>sbatch</tt> command is found [https://slurm.schedmd.com/sbatch.html here].<br />
<br />
<br />
You can query job information using<br />
<br />
<pre><br />
squeue<br />
</pre><br />
<br />
To see only your own jobs, run <br />
<br />
<pre><br />
squeue -u <userid><br />
</pre><br />
<br />
Once your job is running, SLURM creates a file usually named <tt>slurm<jobid>.out</tt> in the directory from where you issued the <tt>sbatch</tt> command. This contains the console output from your job. You can monitor the output of your job by using the <tt>tail -f <file></tt> command.<br />
<br />
<br />
To cancel a job use<br />
<br />
<pre><br />
scancel $JOBID<br />
</pre><br />
<br />
=== Longer jobs ===<br />
<br />
If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.<br />
<br />
=== Interactive ===<br />
<br />
For an interactive session use<br />
<br />
<pre><br />
salloc --gres=gpu:4<br />
</pre><br />
<br />
After executing this command, you may have to wait in the queue until a system is available.<br />
<br />
More information about the <tt>salloc</tt> command is [https://slurm.schedmd.com/salloc.html here].<br />
<br />
=== Automatic Re-submission and Job Dependencies ===<br />
<br />
Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation. <br />
<br />
<pre><br />
#!/bin/bash <br />
<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
: ${job_number:="1"} # set job_nubmer to 1 if it is undefined<br />
job_number_max=3<br />
<br />
echo "hi from ${SLURM_JOB_ID}"<br />
<br />
#RUN JOB HERE<br />
<br />
<br />
# SUBMIT NEXT JOB<br />
if [[ ${job_number} -lt ${job_number_max} ]]<br />
then<br />
(( job_number++ ))<br />
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")<br />
echo "submitted ${next_jobid}"<br />
fi<br />
<br />
sleep 15<br />
<br />
echo "${SLURM_JOB_ID} done"<br />
<br />
</pre><br />
===Packing single-GPU jobs within one SLURM job submission===<br />
Jobs are scheduled by node (4 GPUs) on SOSCIP GPU cluster. If user's code/program cannot utilize all 4 GPUs, user can use GNU Parallel tool to pack 4 or more single-GPU jobs into one SLURM job. Below is an example of submitting 4 single-GPU python codes within one job: (When using GNU parallel for a publication please cite as per '''''parallel --citation''''') <br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
module load gnu-parallel/20180422<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
parallel -a jobname-params.input --colsep ' ' -j 4 'CUDA_VISIBLE_DEVICES=$(( {%} - 1 )) numactl -N $(( ({%} -1) / 2 )) python {1} {2} {3} &> jobname-{#}.out'<br />
</pre><br />
The jobname-params.input file contains:<br />
<pre><br />
code-1.py --param1=a --param2=b<br />
code-2.py --param1=c --param2=d<br />
code-3.py --param1=e --param2=f<br />
code-4.py --param1=g --param2=h<br />
</pre><br />
*In the above example, GNU Parallel tool will read '''jobname-params.input''' file and separate parameters. Each row in the input file has to contain exact 3 parameters to '''python'''. code-N.py is also considered as a parameter. User can change parameter number in the '''parallel''' command ({1} {2} {3}...). <br />
*'''"-j 4"''' flag limits the max number of jobs to be 4. User can have more rows in the input file, but GNU Parallel tool only executes maximum of 4 at the same time. <br />
*'''"CUDA_VISIBLE_DEVICES=$(( {%} - 1 ))"''' will set one GPU for each job. '''"numactl -N $(( ({%} -1) / 2 ))"''' will bind 2 jobs on CPU socket 0, other 2 jobs on socket 1. {%} is job slot which will be translated to 1 or 2 or 3 or 4 in this case.<br />
*Outputs will be jobname-1.out, jobname-2.out,jobname-3.out,jobname-4.out... {#} is job number which will be translated to the row number in the input file.<br />
<br />
== Software Installed ==<br />
<br />
=== IBM PowerAI ===<br />
<br />
The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.<br />
<br />
===GNU Compilers ===<br />
<br />
System default compiler is GCC/5.4.0. More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advance Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:<br />
<br />
Advance Toolchain V10.0<br />
<pre><br />
module load gcc/6.4.1<br />
</pre><br />
<br />
Advance Toolchain V11.0<br />
<pre><br />
module load gcc/7.3.1<br />
</pre><br />
<br />
More information about the IBM Advance Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]<br />
<br />
=== IBM XL Compilers ===<br />
<br />
To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run<br />
<br />
<pre><br />
module load xlc/13.1.5<br />
module load xlf/15.1.5<br />
</pre><br />
<br />
IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.<br />
<br />
Information about the IBM XL Compilers can be found at the following links:<br />
<br />
[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]<br />
<br />
[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]<br />
<br />
=== NVIDIA GPU Driver ===<br />
<br />
The current NVIDIA driver version is 396.26<br />
<br />
=== CUDA ===<br />
<br />
The current installed CUDA Tookits is are version 8.0, 9.0 and 9.1.<br />
<br />
<pre><br />
module load cuda/8.0<br />
or <br />
module load cuda/9.0<br />
or <br />
module load cuda/9.1<br />
or<br />
module load cuda/9.2<br />
</pre><br />
<br />
<br />
The CUDA driver is installed locally, however the CUDA Toolkit is installed in:<br />
<br />
<pre><br />
/usr/local/cuda-8.0<br />
/usr/local/cuda-9.0<br />
/usr/local/cuda-9.1<br />
/usr/local/cuda-9.2<br />
</pre><br />
<br />
Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.2</tt> directory.<br />
<br />
Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]<br />
<br />
=== OpenMPI ===<br />
<br />
Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.<br />
<br />
<pre><br />
$ module load openmpi/2.1.1-gcc-5.4.0<br />
$ module load openmpi/2.1.1-XL-13_15.1.5<br />
</pre><br />
<br />
== Other Software ==<br />
<br />
Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).<br />
<br />
In the following subsections are instructions for installing several common software packages.<br />
<br />
=== Anaconda (Python) ===<br />
<br />
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.<br />
<br />
Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]<br />
<br />
NOTE: Be sure to download the '''Power8''' installer.<br />
<br />
TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda<br />
<br />
=== cuDNN ===<br />
The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN accelerates widely used deep learning frameworks, including Caffe2, MATLAB, Microsoft Cognitive Toolkit, TensorFlow, Theano, and PyTorch. If a specific version of cuDNN is needed, user can download from https://developer.nvidia.com/cudnn and choose '''"cuDNN [VERSION] Library for Linux (Power8/Power9)"'''. <br />
<br />
The default cuDNN installed on the system is version 6 with CUDA-8 from IBM PowerAI. More recent cuDNN versions are installed as modules:<br />
<pre><br />
cudnn/cuda9.0/7.0.5<br />
</pre><br />
<br />
=== Keras ===<br />
<br />
Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.<br />
<br />
*The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command. Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.<br />
<br />
*Keras can also be installed into a Python virtual environment by using '''pip'''. User can install optimized scipy (built with OpenBLAS) before installing Keras.<br />
In a virtual environment (python2.7 as example):<br />
<pre><br />
pip install /scinet/sgc/Libraries/scipy/scipy-1.1.0-cp27-cp27mu-linux_ppc64le.whl<br />
pip install keras<br />
</pre><br />
<br />
=== NumPy/SciPy (built with OpenBLAS) ===<br />
<br />
Optimized NumPy and SciPy are provided as Python wheels located in '''/scinet/sgc/Libraries/numpy''' and '''/scinet/sgc/Libraries/scipy''' and can be installed by '''pip'''. Please uninstall old numpy/scipy before installing the new ones.<br />
<br />
=== PyTorch ===<br />
<br />
PyTorch is the Python implementation of the Torch framework for deep learning. <br />
<br />
It is suggested that you use PyTorch within Anaconda.<br />
<br />
There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.<br />
<br />
Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]<br />
<br />
Before building PyTorch, make sure to load cuda by running <br />
<br />
<pre><br />
module load cuda/8.0<br />
</pre><br />
<br />
NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.<br />
<br />
=== TensorFlow (new versions and python3) ===<br />
<br />
The TensorFlow which is included in PowerAI may not be the most recent version. Newer versions of TensorFlow are provided as prebuilt Python Wheels that users can use '''pip''' to install under user space. Custom Python wheels are stored in '''/scinet/sgc/Applications/TensorFlow_wheels'''. It is highly recommended to install custom TensorFlow wheels into a Python virtual environment.<br />
<br />
====Installing with Python2.7:====<br />
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;"><br />
* Create a virtual environment '''tensorflow-1.8-py2''' with packages installed with system:<br />
<pre><br />
virtualenv --python=python2.7 --system-site-packages tensorflow-1.8-py2<br />
</pre><br />
* Activate virtual environment:<br />
<pre><br />
source tensorflow-1.8-py2/bin/activate<br />
</pre><br />
* Install TensorFlow into the virtual environment: (A custom Numpy built with OpenBLAS library can be installed)<br />
<pre><br />
pip install --upgrade --force-reinstall /scinet/sgc/Libraries/numpy/numpy-1.14.3-cp27-cp27mu-linux_ppc64le.whl<br />
pip install /scinet/sgc/Applications/TensorFlow_wheels/tensorflow-1.8.0-cp27-cp27mu-linux_ppc64le.whl<br />
</pre><br />
</div><br />
<br />
====Installing with Python3.5:====<br />
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;"><br />
* Create a virtual environment '''tensorflow-1.8-py3''' with packages installed with system:<br />
<pre><br />
virtualenv --python=python3.5 --system-site-packages tensorflow-1.8-py3<br />
</pre><br />
* Activate virtual environment:<br />
<pre><br />
source tensorflow-1.8-py3/bin/activate<br />
</pre><br />
* Install TensorFlow into the virtual environment: (A custom Numpy built with OpenBLAS library can be installed)<br />
<pre><br />
pip3 install --upgrade --force-reinstall /scinet/sgc/Libraries/numpy/numpy-1.14.3-cp35-cp35m-linux_ppc64le.whl<br />
pip3 install /scinet/sgc/Applications/TensorFlow_wheels/tensorflow-1.8.0-cp35-cp35m-linux_ppc64le.whl<br />
</pre><br />
</div><br />
<br />
====Submitting jobs====<br />
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;"><br />
The above myjob.script file needs to be modified to run custom TensorFlow. '''cuda/9.0''' and '''cudnn/cuda9.0/7.0.5''' modules need to be loaded. Virtual environment needs to be activated.<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
module purge<br />
module load cuda/9.0 cudnn/cuda9.0/7.0.5<br />
source tensorflow-1.8-py2/bin/activate #change this to the location where virtual environment is created<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
python code.py<br />
</pre><br />
</div><br />
<br />
== LINKS ==<br />
<br />
[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]<br />
<br />
== DOCUMENTATION ==<br />
<br />
# GPU Cluster Introduction: [[Media:GPU_Training_01.pdf|SOSCIP GPU Platform]]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=SOSCIP_GPU&diff=9374SOSCIP GPU2018-08-09T17:27:54Z<p>Rzon: </p>
<hr />
<div>__NOTOC__<br />
<br />
{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:S882lc.png|center|300px|thumb]]<br />
|name=SOSCIP GPU <br />
|installed=September 2017<br />
|operatingsystem= Ubuntu 16.04 le <br />
|loginnode= sgc01 <br />
|nnodes= 14x Power 8 with 4x NVIDIA P100<br />
|rampernode=512 GB<br />
|corespernode= 2 x 10core (20 physical, 160 SMT)<br />
|interconnect=Infiniband EDR <br />
|vendorcompilers=xlc/xlf, nvcc<br />
}}<br />
<br />
== SOSCIP ==<br />
<br />
The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].<br />
<br />
== Support Email ==<br />
<br />
Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.<br />
<br />
== Specifications==<br />
<br />
The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.<br />
<br />
== Access and Login ==<br />
<br />
In order to obtain access to the system, you must request access to the SOSCIP GPU Platform. Instructions will have been sent to your sponsoring faculty member via E-mail at the beginning of your SOSCIP project.<br />
<br />
Access to the SOSCIP GPU Platform is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>''' via ssh. Your user name and password is the same as it is for SciNet systems.<br />
<br />
== Filesystem ==<br />
<br />
The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.<br />
<br />
== Job Submission ==<br />
<br />
The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.<br />
<br />
<pre><br />
$ sbatch myjob.script<br />
</pre><br />
<br />
Where myjob.script is <br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
hostname<br />
nvidia-smi<br />
</pre><br />
<br />
More information about the <tt>sbatch</tt> command is found [https://slurm.schedmd.com/sbatch.html here].<br />
<br />
<br />
You can query job information using<br />
<br />
<pre><br />
squeue<br />
</pre><br />
<br />
To see only your own jobs, run <br />
<br />
<pre><br />
squeue -u <userid><br />
</pre><br />
<br />
Once your job is running, SLURM creates a file usually named <tt>slurm<jobid>.out</tt> in the directory from where you issued the <tt>sbatch</tt> command. This contains the console output from your job. You can monitor the output of your job by using the <tt>tail -f <file></tt> command.<br />
<br />
<br />
To cancel a job use<br />
<br />
<pre><br />
scancel $JOBID<br />
</pre><br />
<br />
=== Longer jobs ===<br />
<br />
If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.<br />
<br />
=== Interactive ===<br />
<br />
For an interactive session use<br />
<br />
<pre><br />
salloc --gres=gpu:4<br />
</pre><br />
<br />
After executing this command, you may have to wait in the queue until a system is available.<br />
<br />
More information about the <tt>salloc</tt> command is [https://slurm.schedmd.com/salloc.html here].<br />
<br />
=== Automatic Re-submission and Job Dependencies ===<br />
<br />
Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation. <br />
<br />
<pre><br />
#!/bin/bash <br />
<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
: ${job_number:="1"} # set job_nubmer to 1 if it is undefined<br />
job_number_max=3<br />
<br />
echo "hi from ${SLURM_JOB_ID}"<br />
<br />
#RUN JOB HERE<br />
<br />
<br />
# SUBMIT NEXT JOB<br />
if [[ ${job_number} -lt ${job_number_max} ]]<br />
then<br />
(( job_number++ ))<br />
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")<br />
echo "submitted ${next_jobid}"<br />
fi<br />
<br />
sleep 15<br />
<br />
echo "${SLURM_JOB_ID} done"<br />
<br />
</pre><br />
===Packing single-GPU jobs within one SLURM job submission===<br />
Jobs are scheduled by node (4 GPUs) on SOSCIP GPU cluster. If user's code/program cannot utilize all 4 GPUs, user can use GNU Parallel tool to pack 4 or more single-GPU jobs into one SLURM job. Below is an example of submitting 4 single-GPU python codes within one job: (When using GNU parallel for a publication please cite as per '''''parallel --citation''''') <br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
module load gnu-parallel/20180422<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
parallel -a jobname-params.input --colsep ' ' -j 4 'CUDA_VISIBLE_DEVICES=$(( {%} - 1 )) numactl -N $(( ({%} -1) / 2 )) python {1} {2} {3} &> jobname-{#}.out'<br />
</pre><br />
The jobname-params.input file contains:<br />
<pre><br />
code-1.py --param1=a --param2=b<br />
code-2.py --param1=c --param2=d<br />
code-3.py --param1=e --param2=f<br />
code-4.py --param1=g --param2=h<br />
</pre><br />
*In the above example, GNU Parallel tool will read '''jobname-params.input''' file and separate parameters. Each row in the input file has to contain exact 3 parameters to '''python'''. code-N.py is also considered as a parameter. User can change parameter number in the '''parallel''' command ({1} {2} {3}...). <br />
*'''"-j 4"''' flag limits the max number of jobs to be 4. User can have more rows in the input file, but GNU Parallel tool only executes maximum of 4 at the same time. <br />
*'''"CUDA_VISIBLE_DEVICES=$(( {%} - 1 ))"''' will set one GPU for each job. '''"numactl -N $(( ({%} -1) / 2 ))"''' will bind 2 jobs on CPU socket 0, other 2 jobs on socket 1. {%} is job slot which will be translated to 1 or 2 or 3 or 4 in this case.<br />
*Outputs will be jobname-1.out, jobname-2.out,jobname-3.out,jobname-4.out... {#} is job number which will be translated to the row number in the input file.<br />
<br />
== Software Installed ==<br />
<br />
=== IBM PowerAI ===<br />
<br />
The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.<br />
<br />
===GNU Compilers ===<br />
<br />
System default compiler is GCC/5.4.0. More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advance Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:<br />
<br />
Advance Toolchain V10.0<br />
<pre><br />
module load gcc/6.4.1<br />
</pre><br />
<br />
Advance Toolchain V11.0<br />
<pre><br />
module load gcc/7.3.1<br />
</pre><br />
<br />
More information about the IBM Advance Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]<br />
<br />
=== IBM XL Compilers ===<br />
<br />
To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run<br />
<br />
<pre><br />
module load xlc/13.1.5<br />
module load xlf/15.1.5<br />
</pre><br />
<br />
IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.<br />
<br />
Information about the IBM XL Compilers can be found at the following links:<br />
<br />
[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]<br />
<br />
[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]<br />
<br />
=== NVIDIA GPU Driver ===<br />
<br />
The current NVIDIA driver version is 396.26<br />
<br />
=== CUDA ===<br />
<br />
The current installed CUDA Tookits is are version 8.0, 9.0 and 9.1.<br />
<br />
<pre><br />
module load cuda/8.0<br />
or <br />
module load cuda/9.0<br />
or <br />
module load cuda/9.1<br />
or<br />
module load cuda/9.2<br />
</pre><br />
<br />
<br />
The CUDA driver is installed locally, however the CUDA Toolkit is installed in:<br />
<br />
<pre><br />
/usr/local/cuda-8.0<br />
/usr/local/cuda-9.0<br />
/usr/local/cuda-9.1<br />
/usr/local/cuda-9.2<br />
</pre><br />
<br />
Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.2</tt> directory.<br />
<br />
Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]<br />
<br />
=== OpenMPI ===<br />
<br />
Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.<br />
<br />
<pre><br />
$ module load openmpi/2.1.1-gcc-5.4.0<br />
$ module load openmpi/2.1.1-XL-13_15.1.5<br />
</pre><br />
<br />
== Other Software ==<br />
<br />
Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).<br />
<br />
In the following subsections are instructions for installing several common software packages.<br />
<br />
=== Anaconda (Python) ===<br />
<br />
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.<br />
<br />
Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]<br />
<br />
NOTE: Be sure to download the '''Power8''' installer.<br />
<br />
TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda<br />
<br />
=== cuDNN ===<br />
The NVIDIA CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN accelerates widely used deep learning frameworks, including Caffe2, MATLAB, Microsoft Cognitive Toolkit, TensorFlow, Theano, and PyTorch. If a specific version of cuDNN is needed, user can download from https://developer.nvidia.com/cudnn and choose '''"cuDNN [VERSION] Library for Linux (Power8/Power9)"'''. <br />
<br />
The default cuDNN installed on the system is version 6 with CUDA-8 from IBM PowerAI. More recent cuDNN versions are installed as modules:<br />
<pre><br />
cudnn/cuda9.0/7.0.5<br />
</pre><br />
<br />
=== Keras ===<br />
<br />
Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.<br />
<br />
*The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command. Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.<br />
<br />
*Keras can also be installed into a Python virtual environment by using '''pip'''. User can install optimized scipy (built with OpenBLAS) before installing Keras.<br />
In a virtual environment (python2.7 as example):<br />
<pre><br />
pip install /scinet/sgc/Libraries/scipy/scipy-1.1.0-cp27-cp27mu-linux_ppc64le.whl<br />
pip install keras<br />
</pre><br />
<br />
=== NumPy/SciPy (built with OpenBLAS) ===<br />
<br />
Optimized NumPy and SciPy are provided as Python wheels located in '''/scinet/sgc/Libraries/numpy''' and '''/scinet/sgc/Libraries/scipy''' and can be installed by '''pip'''. Please uninstall old numpy/scipy before installing the new ones.<br />
<br />
=== PyTorch ===<br />
<br />
PyTorch is the Python implementation of the Torch framework for deep learning. <br />
<br />
It is suggested that you use PyTorch within Anaconda.<br />
<br />
There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.<br />
<br />
Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]<br />
<br />
Before building PyTorch, make sure to load cuda by running <br />
<br />
<pre><br />
module load cuda/8.0<br />
</pre><br />
<br />
NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.<br />
<br />
=== TensorFlow (new versions and python3) ===<br />
<br />
The TensorFlow which is included in PowerAI may not be the most recent version. Newer versions of TensorFlow are provided as prebuilt Python Wheels that users can use '''pip''' to install under user space. Custom Python wheels are stored in '''/scinet/sgc/Applications/TensorFlow_wheels'''. It is highly recommended to install custom TensorFlow wheels into a Python virtual environment.<br />
<br />
====Installing with Python2.7:====<br />
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;"><br />
* Create a virtual environment '''tensorflow-1.8-py2''' with packages installed with system:<br />
<pre><br />
virtualenv --python=python2.7 --system-site-packages tensorflow-1.8-py2<br />
</pre><br />
* Activate virtual environment:<br />
<pre><br />
source tensorflow-1.8-py2/bin/activate<br />
</pre><br />
* Install TensorFlow into the virtual environment: (A custom Numpy built with OpenBLAS library can be installed)<br />
<pre><br />
pip install --upgrade --force-reinstall /scinet/sgc/Libraries/numpy/numpy-1.14.3-cp27-cp27mu-linux_ppc64le.whl<br />
pip install /scinet/sgc/Applications/TensorFlow_wheels/tensorflow-1.8.0-cp27-cp27mu-linux_ppc64le.whl<br />
</pre><br />
</div><br />
<br />
====Installing with Python3.5:====<br />
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;"><br />
* Create a virtual environment '''tensorflow-1.8-py3''' with packages installed with system:<br />
<pre><br />
virtualenv --python=python3.5 --system-site-packages tensorflow-1.8-py3<br />
</pre><br />
* Activate virtual environment:<br />
<pre><br />
source tensorflow-1.8-py3/bin/activate<br />
</pre><br />
* Install TensorFlow into the virtual environment: (A custom Numpy built with OpenBLAS library can be installed)<br />
<pre><br />
pip3 install --upgrade --force-reinstall /scinet/sgc/Libraries/numpy/numpy-1.14.3-cp35-cp35m-linux_ppc64le.whl<br />
pip3 install /scinet/sgc/Applications/TensorFlow_wheels/tensorflow-1.8.0-cp35-cp35m-linux_ppc64le.whl<br />
</pre><br />
</div><br />
<br />
====Submitting jobs====<br />
<div class="toccolours mw-collapsible mw-collapsed" style="overflow:auto;"><br />
The above myjob.script file needs to be modified to run custom TensorFlow. '''cuda/9.0''' and '''cudnn/cuda9.0/7.0.5''' modules need to be loaded. Virtual environment needs to be activated.<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
module purge<br />
module load cuda/9.0 cudnn/cuda9.0/7.0.5<br />
source tensorflow-1.8-py2/bin/activate #change this to the location where virtual environment is created<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
python code.py<br />
</pre><br />
</div><br />
<br />
== LINKS ==<br />
<br />
[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]<br />
<br />
== DOCUMENTATION ==<br />
<br />
# GPU Cluster Introduction: [[Media:GPU_Training_01.pdf|SOSCIP GPU Platform]]</div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ&diff=9373BGQ2018-08-09T17:27:41Z<p>Rzon: </p>
<hr />
<div>{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
{{Infobox Computer<br />
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]<br />
|name=Blue Gene/Q (BGQ)<br />
|installed=Aug 2012, Nov 2014<br />
|operatingsystem= RH6.3, CNK (Linux) <br />
|loginnode= bgqdev-fen1<br />
|nnodes= 4096 nodes (65,536 cores)<br />
|rampernode=16 GB <br />
|corespernode=16 (64 threads)<br />
|interconnect=5D Torus (jobs), QDR Infiniband (I/O) <br />
|vendorcompilers= bgxlc, bgxlf<br />
|queuetype=Loadleveler<br />
}}<br />
<br />
==System Status==<br />
<br />
The current BGQ system status can be found on the wiki's [[Main Page]].<br />
<br />
==SOSCIP & LKSAVI==<br />
<br />
The BGQ is a Southern Ontario Smart Computing<br />
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the<br />
University of Toronto's SciNet HPC facility. The SOSCIP <br />
multi-university/industry consortium is funded by the Ontario Government <br />
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].<br />
<br />
A half-rack of BlueGene/Q (8,192 cores) was purchased by the [http://likashingvirology.med.ualberta.ca/ Li Ka Shing Institute of Virology] at the University of Alberta in late fall 2014 and integrated into the existing BGQ system.<br />
<br />
The combined 4 rack system is the fastest Canadian supercomputer on the [http://top500.org/ top 500], currently at the 120th place (Nov 2015).<br />
<br />
== Support Email ==<br />
<br />
Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.<br />
<br />
==Specifications==<br />
<br />
BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK ('''C'''ompute '''N'''ode '''K'''ernel). The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet's BGQ consists of 8 mdiplanes (four-racks) totalling 65,536 cores and 64TB of RAM.<br />
<br />
[[Image:BlueGeneQHardware2.png |center]]<br />
<br />
=== 5D Torus Network ===<br />
<br />
The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.<br />
<br />
{|border="1" cellspacing="0" cellpadding="2"<br />
| '''Node Boards '''<br />
| '''Compute Nodes'''<br />
| '''Cores'''<br />
| '''Torus Dimensions'''<br />
|-<br />
| 1<br />
| 32<br />
| 512<br />
| 2x2x2x2x2<br />
|-<br />
| 2 (adjacent pairs)<br />
| 64<br />
| 1024<br />
| 2x2x4x2x2<br />
|-<br />
| 4 (quadrants)<br />
| 128<br />
| 2048<br />
| 2x2x4x4x2<br />
|-<br />
| 8 (halves)<br />
| 256<br />
| 4096<br />
| 4x2x4x4x2<br />
|-<br />
| 16 (midplane)<br />
| 512<br />
| 8192<br />
| 4x4x4x4x2<br />
|-<br />
| 32 (1 rack)<br />
| 1024<br />
| 16384<br />
| 4x4x4x8x2 <br />
|-<br />
| 64 (2 racks)<br />
| 2048<br />
| 32768<br />
| 4x4x8x8x2<br />
|-<br />
| 96 (3 racks)<br />
| 3072<br />
| 49152<br />
| 4x4x12x8x2<br />
|-<br />
| 128 (4 racks)<br />
| 4096<br />
| 65536<br />
| 8x4x8x8x2<br />
|}<br />
<br />
== Login/Devel Node ==<br />
<br />
The development node is '''bgqdev-fen1''' which one can login to from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside using '''bgqdev.scinet.utoronto.ca''', e.g.<br />
<pre><br />
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X<br />
</pre><br />
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics.<br/><br />
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].<br />
<br />
These development node is a Power7 machines running Linux which serve as the compilation and submission host for the BGQ. Programs are cross-compiled for the BGQ on this node and then submitted to the queue using loadleveler.<br />
<br />
===Modules and Environment Variables===<br />
<br />
To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.<br />
<br />
A list of the installed software can be seen on the system by typing <br />
<pre><br />
$ module avail<br />
</pre><br />
<br />
To load a module (for example, the default version of the intel compilers)<br />
<pre><br />
$ module load vacpp<br />
</pre><br />
To unload a module<br />
<pre><br />
$ module unload vacpp<br />
</pre><br />
To unload all modules<br />
<pre><br />
$ module purge<br />
</pre><br />
<br />
These commands can go in your .bashrc files to make sure you are using the correct packages.<br />
<br />
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention<br />
<pre><br />
$SCINET_[short-module-name]_BASE<br />
$SCINET_[short-module-name]_LIB<br />
$SCINET_[short-module-name]_INC<br />
</pre><br />
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.<br />
<br />
So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>. <br />
<br />
Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.<br />
<br />
If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).<br />
<br />
=== Compilers ===<br />
<br />
The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce<br />
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',<br />
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively. <br />
<br />
Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading<br />
the '''mpich2''' module. <br />
<pre><br />
module load mpich2<br />
</pre><br />
<br />
It is recommended to use at least the following flags when compiling and linking<br />
<pre><br />
-O3 -qarch=qp -qtune=qp<br />
</pre><br />
<br />
If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.<br />
<br />
== ION/Devel Nodes ==<br />
<br />
There are also bgq native development nodes named '''bgqdev-ion[01-24]''' which one can login to directly, i.e. ssh, from '''bgqdev-fen1'''. These nodes are extra I/O nodes that are essentially the same as the BGQ compute nodes with the exception that they run a full RedHat Linux and have an infiniband interface providing direct network access. Unlike the regular development node, '''bgqdev-fen1''', which is Power7, this node has the same BGQ A2 processor, and thus cross compilations are not required which can make building some software easier. <br />
<br />
'''NOTE''': BGQ MPI jobs can be compiled on these nodes, however can not be run locally as the mpich2 is setup for the BGQ network and thus will fail on these nodes.<br />
<br />
== Job Submission ==<br />
<br />
As the BlueGene/Q architecture is different from the development nodes, you cannot run applications intended/compiled for the BGQ on the devel nodes. The only way to run (or even test) your program is to submit a job to the BGQ. Jobs are submitted as scripts through loadleveler. That script must then use '''runjob''' to start the job, which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below. <br />
<br />
=== runjob ===<br />
<br />
All BGQ runs are launched using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. There are two ways to get a block. One way is to use a 30-minute 'debugjob' session (more about that below). The other, more common case, is using a job script submitted and are running using loadleveler. Inside the job script, this block is set for you, and you do not have to specify the block name. For example, if your loadleveler job script requests 64 nodes, each with 16 cores (for a total of 1024 cores), from within that job script, you can run a job with 16 processes per node and 1024 total processes with<br />
<pre><br />
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in<br />
</pre><br />
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.<br />
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. <br />
<br />
<!-- (Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>") --><br />
<br />
runjob flags are shown with<br />
<pre><br />
runjob -h<br />
</pre><br />
<br />
a particularly useful one is<br />
<br />
<pre><br />
--verbose #<br />
</pre><br />
<br />
where # is from 1-7 which can be helpful in debugging an application.<br />
<br />
=== How to set ranks-per-node ===<br />
<br />
There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per core, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:<br />
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)<br />
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.<br />
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.<br />
<br />
Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.<br />
<br />
=== Queue Limits ===<br />
<br />
The maximum wall_clock_limit is 24 hours. Official SOSCIP project jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.<br />
<br />
A 64 node block is reserved for development and interactive testing for 16 hours, from 8AM to midnight, everyday including weekends. While you can still reserve an interactive block from midnight to 8AM, the priority is given to batch jobs at that time interval in order to keep the machine usage as high as possible. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.<br />
<br />
<!-- We need to recover this functionality again. At the moment it doesn't work<br />
=== BACKFILL scheduling ===<br />
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command <span style="color: red;font-weight: bold;">llAvailableResources</span> gives for example :<br />
<pre><br />
On the Devel system : only a debugjob can start immediately<br />
<br />
On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min <br />
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min <br />
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min <br />
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min<br />
</pre><br />
--><br />
<br />
=== Batch Jobs ===<br />
<br />
Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.<br />
<br />
The parameter <span style="font-weight: bold;">bg_size</span> can only be equal to 64, 128, 256, 512, 1024 and 2048.<br />
<br />
<span style="font-weight: bold;">np</span> &le; ranks-per-node * bg_size<br />
<br />
ranks-per-node &le; np<br />
<br />
(ranks-per-node * OMP_NUM_THREADS ) &le; 64 <br />
<br />
np : number of MPI processes<br />
<br />
ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64<br />
<br />
OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Launch all BGQ jobs using runjob<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
</pre><br />
<br />
To submit to the queue use <br />
<br />
<pre><br />
llsubmit myscript.sh<br />
</pre><br />
=== Steps ( Job dependency) ===<br />
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ step_name = step1 <br />
# @ queue<br />
# Launch the first step :<br />
if [ $LOADL_STEP_NAME = "step1" ]; then<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
fi<br />
<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ step_name = step2 <br />
# @ dependency = step1 == 0 <br />
# @ queue<br />
# Launch the second step if the first one has returned 0 (done successfully) :<br />
if [ $LOADL_STEP_NAME = "step2" ]; then<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
fi<br />
<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ step_name = step3 <br />
# @ dependency = step2 == 0 <br />
# @ queue<br />
# Launch the third step if the second one has returned 0 (done successfully) :<br />
if [ $LOADL_STEP_NAME = "step3" ]; then<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
fi<br />
<br />
</pre><br />
<br />
=== Monitoring Jobs ===<br />
<br />
To see running jobs<br />
<br />
<pre><br />
llq2<br />
</pre><br />
<br />
or<br />
<br />
<pre><br />
llq -b<br />
</pre><br />
<br />
<br />
to cancel a job use<br />
<br />
<pre><br />
llcancel JOBID<br />
</pre><br />
<br />
and to look at details of the bluegene resources use<br />
<br />
<pre><br />
llbgstatus -M all<br />
</pre><br />
<br />
'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''<br />
<br />
=== Monitoring Stats ===<br />
<br />
Use llbgstats to monitor your own stats and/or your group stats. PIs can also print their (current) monthly report.<br />
<pre><br />
llbgstats -h<br />
</pre><br />
<br />
=== Interactive Use / Debugging ===<br />
<br />
As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes. <br />
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs, <br />
however an interactive session is typically beneficial when debugging and developing. As such a <br />
script has been written to allow a session in which runjob can be run interactively. The script<br />
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on<br />
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on <br />
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section. <br />
<br />
<pre><br />
[user@bgqdev-fen1]$ debugjob<br />
<br />
[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags<br />
<br />
[user@bgqdev-fen1]$ exit<br />
</pre><br />
<br />
For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.<br />
<br />
Note: when running a job under ddt, you'll need to add "<tt>--ranks-per-node=X</tt>" to the "runjob arguments" field.<br />
<br />
Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables <br />
<pre><br />
$ export BG_PGM_LAUNCHER=yes<br />
$ export RUNJOB_NP=1<br />
</pre><br />
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.<br />
<br />
<br />
debugjob session with an executable implicitly calls runjob with 1 mpi task :<br />
<pre><br />
debugjob -i<br />
**********************************************************<br />
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and <br />
LL14040718574824 for 30 minutes with 64 NODES (1024 cores). <br />
IMPLICIT MODE: running an executable implicitly calls runjob<br />
with 1 mpi task<br />
Exit shell when finished. <br />
**********************************************************<br />
</pre><br />
<br />
=== Sub-block jobs ===<br />
<br />
BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used. <br />
<br />
Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.<br />
<br />
Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER. <br />
<source lang="bash"><br />
#!/bin/bash<br />
# @ job_name = bgsubblock<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job SUBBLOCK "<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue<br />
<br />
# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}<br />
# with size of subblocks in nodes (ie similiar to bg_size)<br />
<br />
# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)<br />
source subblocks 4<br />
<br />
# 16 jobs of 4 each<br />
for (( i=0; i < 16 ; i++)); do<br />
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &<br />
done<br />
wait<br />
</source><br />
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.<br />
<br />
Let us know if you run into any issues with this technique, please contact bgq-support for help.<br />
<br />
== Filesystem ==<br />
<br />
The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems). <br />
<br />
{|border="1" cellpadding="10" cellspacing="0"<br />
|-<br />
! {{Hl2}} | file system <br />
! {{Hl2}} | purpose <br />
! {{Hl2}} | user quota <br />
! {{Hl2}} | backed up<br />
! {{Hl2}} | purged<br />
|- <br />
| /home<br />
| development<br />
| 50 GB<br />
| yes<br />
| never<br />
|-<br />
| /scratch<br />
| computation<br />
| first of (20 TB ; 1 million files)<br />
| no<br />
| not currently<br />
|}<br />
<br />
===Transfering files===<br />
The BGQ GPFS file system, except for HPSS, is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ. <br />
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do<br />
<pre><br />
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .<br />
</pre><br />
or from a login node you could do<br />
<pre><br />
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:<br />
</pre><br />
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back. <br />
<br />
Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.<br />
<br />
===How much Disk Space Do I have left?===<br />
<br />
The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.<br />
<pre><br />
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]<br />
-h|-?: help<br />
-a: list usages of all members on the group<br />
-u <user>: as another user on your group<br />
-de: include delta information<br />
-plot: create plots of disk usages<br />
</pre><br />
Note that the information on usage and quota is only updated hourly!<br />
<br />
===Bridge to HPSS===<br />
<br />
BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]<br />
<br />
== Software modules installed on the BGQ ==<br />
<br />
{| border="1" cellpadding="10" cellspacing="0"<br />
!{{Hl2}} |Software <br />
!{{Hl2}}| Version<br />
!{{Hl2}}| Comments<br />
!{{Hl2}}| Command/Library<br />
!{{Hl2}}| Module Name<br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Compilers & Development Tools'''''<br />
|-<br />
|IBM fortran compiler<br />
|14.1<br />
|These are cross compilers<br />
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt><br />
|xlf<br />
|-<br />
|IBM c/c++ compilers<br />
|12.1<br />
|These are cross compilers<br />
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt><br />
|vacpp<br />
|-<br />
|MPICH2 MPI library<br />
|1.4.1<br />
|There are 4 versions (see BGQ Applications Development document).<br />
|<tt>mpicc,mpicxx,mpif77,mpif90</tt><br />
|mpich2<br />
|- <br />
| GCC Compiler<br />
| 4.4.6, 4.8.1<br />
| GNU Compiler Collection for BGQ<br>(4.8.1 requires binutils/2.23 to be loaded)<br />
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt><br />
| <tt>bgqgcc</tt><br />
|-<br />
| Clang Compiler<br />
| r217688-20140912, r263698-20160317<br />
| Clang cross-compilers for bgq<br />
| <tt>powerpc64-bgq-linux-clang, powerpc64-bgq-linux-clang++</tt><br />
| <tt>bgclang</tt><br />
|-<br />
| Binutils<br />
| 2.21.1, 2.23<br />
| Cross-compilation utilities<br />
| <tt>addr2line, ar, ld, ...</tt><br />
| <tt>binutils</tt><br />
|-<br />
| CMake <br />
| 2.8.8, 2.8.12.1<br />
| cross-platform, open-source build system<br />
| <tt>cmake</tt><br />
| <tt>cmake</tt><br />
|-<br />
| Git<br />
| 1.9.5<br />
| Revision control system<br />
| <tt>git, gitk</tt><br />
| <tt>git</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''<br />
|-<br />
| [https://www.gnu.org/software/gdb/ gdb]<br />
| 7.2<br />
| GNU Debugger<br />
| <tt>gdb</tt><br />
| <tt>gdb</tt><br />
|-<br />
| [https://www.gnu.org/software/ddd/ ddd]<br />
| 3.3.12<br />
| GNO Data Display Debugger<br />
| <tt>ddd</tt><br />
| <tt>ddd</tt><br />
|- <br />
| [http://www.allinea.com/products/ddt/ DDT]<br />
| 4.1, 4.2, 5.0.1<br />
| Allinea's Distributed Debugging Tool<br />
| <tt>ddt</tt><br />
| <tt>ddt</tt><br />
|- <br />
| [[HPCTW]]<br />
| 1.0<br />
| BGQ MPI and Hardware Counters<br />
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt><br />
| <tt>hptibm</tt><br />
|- <br />
| [[MemP]]<br />
| 1.0.3<br />
| BGQ Memory Stats<br />
| <tt>libmemP.a </tt><br />
| <tt>memP</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''<br />
|-<br />
| HDF5<br />
| 1.8.9-v18<br />
| Scientific data storage and retrieval<br />
| <tt>h5ls, h5diff, ..., libhdf5</tt><br />
| <tt>hdf5/189-v18-serial-xlc*<br/>hdf5/189-v18-mpich2-xlc</tt><br />
|-<br />
| HDF5<br />
| 1.8.12-v18<br />
| Scientific data storage and retrieval<br />
| <tt>h5ls, h5diff, ..., libhdf5</tt><br />
| <tt>hdf5/1812-v18-serial-gcc<br/>hdf5/1812-v18-mpich2-gcc</tt><br />
|-<br />
| NetCDF<br />
| 4.2.1.1<br />
| Scientific data storage and retrieval<br />
| <tt>ncdump,ncgen,libnetcdf</tt><br />
| <tt>netcdf/4.2.1.1-serial-xlc*<br/>netcdf/4.2.1.1-mpich2-xlc</tt><br />
|-<br />
| Parallel NetCDF<br />
| 1.3.1<br />
| Parallel scientific data storage and retrieval using MPI-IO<br />
| <tt>libpnetcdf.a</tt><br />
| <tt>parallel-netcdf</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''<br />
|-<br />
| ESSL<br />
| 5.1<br />
| IBM Engineering and Scientific Subroutine Library (manual below)<br />
| <tt>libesslbg,libesslsmpbg</tt><br />
| <tt>essl</tt><br />
|-<br />
| WSMP<br />
| 15.06.01<br />
| Watson Sparse Matrix Package<br />
| <tt>libpwsmpBGQ.a</tt><br />
| <tt>WSMP</tt><br />
|- <br />
| FFTW<br />
| 2.1.5, 3.3.2, 3.1.2-esslwrapper<br />
| Fast fourier transform <br />
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt><br />
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt><br />
|-<br />
| LAPACK + ScaLAPACK<br />
| 3.4.2 + 2.0.2<br />
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.<br />
| <tt>liblapack, libscalpack</tt><br />
| lapack<br />
|-<br />
| GSL<br />
| 1.15<br />
| GNU Scientific Library<br />
| <tt>libgsl, libgslcblas</tt><br />
| <tt>gsl</tt><br />
|-<br />
| BOOST<br />
| 1.47.0, 1.54, 1.57<br />
| C++ Boost libraries<br />
| <tt>libboost...</tt><br />
| <tt>cxxlibraries/boost</tt><br />
|-<br />
| bzip2 + szip + zlib<br />
| 1.0.6 + 2.1 + 1.2.7<br />
| compression libraries<br />
| <tt>libbz2,libz,libsz</tt><br />
| <tt>compression</tt><br />
|-<br />
| METIS<br />
| 5.0.2<br />
| Serial Graph Partitioning and Fill-reducing Matrix Ordering<br />
| <tt>libmetis</tt><br />
| <tt>metis</tt><br />
|-<br />
| ParMETIS<br />
| 4.0.2<br />
| Parallel graph partitioning and fill-reducing matrix ordering<br />
| <tt>libparmetis</tt><br />
| <tt>parmetis</tt> <br />
|-<br />
| OpenSSL<br />
| 1.0.2 <br />
| General-purpose cryptography library<br />
| <tt>libcrypto, libssl</tt><br />
| <tt>openssl</tt> <br />
|-<br />
| FILTLAN<br />
| 1.0<br />
| The Filtered Lanczos Package <br />
| <tt>libdfiltlan,libdmatkit,libsfiltlan,libsmatkit</tt><br />
| <tt>FILTLAN</tt> <br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''<br />
|-<br />
| [[Python]]<br />
| 2.6.6<br />
| Python programming language<br />
| <tt>/bgsys/tools/Python-2.6/bin/python</tt><br />
| <tt>python</tt><br />
|-<br />
| [[Python]]<br />
| 2.7.3<br />
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1<br />
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt><br />
| <tt>python</tt><br />
|-<br />
| [[Python]]<br />
| 3.2.2<br />
| Python programming language<br />
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt><br />
| <tt>python</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''<br />
|-<br />
| [http://www.abinit.org/ ABINIT]<br />
| 7.10.4<br />
| An atomic-scale simulation software suite<br />
| <tt>abinit</tt><br />
| <tt>abinit</tt><br />
|-<br />
| [http://www.berkeleygw.org/ BerkeleyGW library]<br />
| 1.0.4-2.0.0436<br />
| Computes quasiparticle properties and the optical responses of a large variety of materials<br />
| <tt>libBGW_wfn.a, wfn_rho_vxc_io_m.mod</tt><br />
| <tt>BGW-paratec</tt><br />
|-<br />
| [https://www.cp2k.org/ CP2K]<br />
| 2.3, 2.4, 2.5.1, 2.6.1<br />
| DFT molecular dynamics, MPI <br />
| <tt>cp2k.psmp</tt><br />
| <tt>cp2k</tt><br />
|-<br />
| [http://www.cpmd.org/ CPMD]<br />
| 3.15.3, 3.17.1<br />
| Carr-Parinello molecular dynamics, MPI<br />
| <tt>cpmd.x</tt><br />
| <tt>cpmd</tt><br />
|-<br />
| gnuplot<br />
| 4.6.1<br />
| interactive plotting program to be run on front-end nodes<br />
| <tt>gnuplot</tt><br />
| <tt>gnuplot</tt><br />
|-<br />
| LAMMPS<br />
| Nov 2012/7Dec15/7Dec15-mpi<br />
| Molecular Dynamics <br />
| <tt>lmp_bgq</tt><br />
| <tt>lammps</tt><br />
|-<br />
| NAMD<br />
| 2.9<br />
| Molecular Dynamics <br />
| <tt>namd2</tt><br />
| <tt>namd/2.9-smp</tt><br />
|-<br />
| [http://www.quantum-espresso.org/index.php Quantum Espresso]<br />
| 5.0.3/5.2.1<br />
| Molecular Structure / Quantum Chemistry <br />
| <tt>qe_pw.x, etc</tt><br />
| <tt>espresso</tt><br />
|-<br />
| [[BGQ_OpenFOAM | OpenFOAM]]<br />
| 2.2.0, 2.3.0, 2.4.0, 3.0.1<br />
| Computational Fluid Dynamics<br />
| <tt>icofoam,etc. </tt><br />
| <tt>openfoam/2.2.0, openfoam/2.3.0, openfoam/2.4.0</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Beta Tests'''''<br />
|-<br />
| WATSON API<br />
| beta<br />
| Natural Language Processing<br />
| <tt>watson_beta</tt><br />
| <tt>FEN/WATSON</tt><br />
|-<br />
|}<br />
<br />
=== OpenFOAM on BGQ ===<br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ_OpenFOAM How to use OpenFOAM on BGQ]<br />
<br />
== Python on BlueGene ==<br />
Python 2.7.3 has been installed on BlueGene. To use <span style="color: red;font-weight: bold;">Numpy</span> and <span style="color: red;font-weight: bold;">Scipy</span>, the module <span style="color: red;font-weight: bold;">essl/5.1</span> has to be loaded.<br />
The full python path has to be provided (otherwise the default version is used).<br />
<br />
To use python on BlueGene (from within a job script or a debugjob session):<br />
<source lang="bash"><br />
module load python/2.7.3<br />
##Only if you need numpy/scipy :<br />
module load xlf/14.1 essl/5.1<br />
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py <br />
</source><br />
<br />
If you want to use the mmap python API, you must use it in PRIVATE mode as shown in the bellow example :<br />
<source lang="bash"><br />
import mmap<br />
mm=mmap.mmap(-1,256,mmap.MAP_PRIVATE)<br />
mm.close()<br />
</source><br />
<br />
Alternatively, you can use the mpi4py and h5py modules.<br />
<br />
Also, please read Cython documentation.<br />
<br />
== Documentation ==<br />
#BGQ Day: Intro to Using the BGQ<br/>[[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]]<br/>[[Media:BgqintroUpdatedMarch2015.pdf|Slides (updated in 2015) ]] &nbsp;/ &nbsp; [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]<br />
#BGQ Day: BGQ Hardware Overview<br/>[[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]]<br/>[https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides] &nbsp;/ &nbsp; [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]<br />
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]<br />
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]<br />
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]<br />
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]<br />
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]<br />
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]<br />
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]<br />
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]<br />
# [https://www.ibm.com/support/knowledgecenter/en/SSFJTW_5.1.0/loadl.v5r1_welcome.html IBM LoadLeveler 5.1]<br />
<br />
<!-- PUT IN TRAC !!!<br />
<br />
=== *Manual Block Creation* ===<br />
<br />
To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node <br />
<br />
<pre><br />
bg_console<br />
</pre><br />
<br />
There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the<br />
following command:<br />
<br />
<pre><br />
gen_small_block <blockid> <midplane> <cnodes> <nodeboard> <br />
gen_small_block R00-M0-N03-32 R00-M0 32 N03<br />
</pre><br />
<br />
The block then needs to be booted using:<br />
<br />
<pre><br />
allocate R00-M0-N03-32<br />
</pre><br />
<br />
If those resources are already booted into another block, that block must be freed before the new block can be <br />
allocated.<br />
<pre><br />
free R00-M0-N03<br />
</pre><br />
<br />
There are many other functions in bg_console:<br />
<br />
<pre><br />
help all<br />
</pre><br />
<br />
The BGQ default nomenclature for hardware is as follows:<br />
<br />
<pre><br />
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore<br />
</pre><br />
<br />
So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.<br />
<br />
<br />
--!></div>Rzonhttps://oldwiki.scinet.utoronto.ca/index.php?title=SciNet_User_Support_Library&diff=9365SciNet User Support Library2018-05-15T18:59:32Z<p>Rzon: </p>
<hr />
<div>__NOTOC__<br />
{| style="border-spacing: 8px; width:100%"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:2px solid; background-color:#f6f674; border-radius:5px"|<br />
'''WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to [https://docs.scinet.utoronto.ca https://docs.scinet.utoronto.ca]'''<br />
|}<br />
<br />
<br />
{| style="border-spacing: 8px;"<br />
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|<br />
{{SciNetWiki:System_Alerts}}<br />
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|<br />
==System News==<br />
* April 23, 2018: GPC & Sandy decommissioned.<br />
* April 10, 2018: Niagara commissioned.<br />
* March 20th, 2018: Gravity decommissioned.<br />
* Dec 4, 2017 : scratchtcs decommissioned.<br />
* Nov 28, 2017: The GPC has been reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.<br />
* Sept 29, 2017: The TCS has been decommissioned.<br />
<br />
([[Previous System News]])<br />
|-<br />
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |<br />
<br />
== QuickStart Guides ==<br />
* [[SciNet Command Line Utilities]]<br />
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]<br />
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]<br />
* [[BGQ|BGQ: BlueGene/Q clusters]]<br />
* [[Software_and_Libraries|Software and libraries]]<br />
* [[Data_Management|Data management]]<br />
* [[FAQ | FAQ (frequently asked questions)]]<br />
* [[Acknowledging_SciNet | Acknowledging SciNet]]<br />
* [https://courses.scinet.utoronto.ca SciNet education]<br />
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]<br />
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |<br />
<br />
== Tutorials and Manuals ==<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]<br />
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]<br />
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]<br />
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]<br />
|-<br />
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |<br />
<br />
== What's New On The Wiki ==<br />
<div style='text-align:left;'><br />
<br />
* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.<br />
<br />
* Dec 2014: [[Hdf5]] compilation page updated.<br />
<br />
* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8<br />
<br />
* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].<br />
<br />
* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4<br />
<br />
* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]<br />
<br />
* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.<br />
<br />
* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.<br />
<br />
* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.<br />
<br />
* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).<br />
<br />
</div><br />
Previous new stuff can be found in the [[What's new archive]].<br />
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |<br />
<div style='text-align:left;'><br />
{{SciNetWiki:Community_Portal}}<br />
</div><br />
|}<br />
<br />
{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}<br />
<br />
<!-- [[Old Main Page]] --!></div>Rzon