oldwiki.scinet.utoronto.ca - User contributions [en-gb]

PWC Python

2014-12-02T20:38:01Z

Ljdursi:

This page contains the slides for the PWC Python class.

== Slides ==

* [http://wiki.scinethpc.ca/wiki/images/3/3c/PWCintro.pdf Morning of the first day].
* [http://wiki.scinethpc.ca/wiki/images/2/2d/PWCFirstAfternoon.pdf Afternoon of the first day].

* [[Media:pwcfunctions.pdf | Second day ]]
* [http://support.scinet.utoronto.ca/CourseVideo/pwcfunctions.zip Code ]
** [http://support.scinet.utoronto.ca/~ljdursi/pwc/mapreduce.py mapreduce.py]
** [http://support.scinet.utoronto.ca/~ljdursi/pwc/mapreduce-ans.py mapreduce partial answer]
* [[Media:pwcobjects.pdf | Second day, objects ]]

PWC Python

2014-12-02T18:29:27Z

Ljdursi: /* Slides */

Python

2014-09-16T22:00:24Z

Ljdursi: /* Python on the GPC */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2, 2.7.3, 2.7.5, and 3.3.4 installed, compiled against fast intel math libraries. To load the python modules, type the following commands:

{|
! Version
! Command
|-
|2.7.2
|<tt>module load gcc intel python</tt>
|-
|2.7.3
|<tt>module load gcc intel/13.1.1 python/2.7.3</tt>
|-
|2.7.5
|<tt>module load gcc intel/13.1.1 python/2.7.5</tt>
|-
|3.3.4
|<tt>module load gcc intel/14.0.1 python/3.3.4</tt>
|}

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support. No longer supported.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded.
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded.
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module. No longer supported.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
|
|
|
| Python interface to the svn version control system.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 5.1
| Enables easy installation of new python modules
|-
| [http://pandas.pydata.org/ pandas]
| 0.13.0
| 0.13.0
| 0.13.0
| 0.14.1
| high-performance, easy-to-use data structures and data analysis tools.
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T21:42:48Z

Ljdursi: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support. No longer supported.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded.
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded.
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module. No longer supported.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
|
|
|
| Python interface to the svn version control system.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 5.1
| Enables easy installation of new python modules
|-
| [http://pandas.pydata.org/ pandas]
| 0.13.0
| 0.13.0
| 0.13.0
| 0.14.1
| high-performance, easy-to-use data structures and data analysis tools.
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T21:42:14Z

Ljdursi: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support. No longer supported.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded.
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded.
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module. No longer supported.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
|
|
|
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 5.1
| Enables easy installation of new python modules
|-
| [http://pandas.pydata.org/ pandas]
| 0.13.0
| 0.13.0
| 0.13.0
| 0.14.1
| high-performance, easy-to-use data structures and data analysis tools.
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Using Paraview

2014-09-10T20:04:37Z

Ljdursi: /* Connect Client and Server */

[http://www.paraview.org/ ParaView] is a powerful, parallel, client-server based visualization system that allows you to use SciNet's GPC nodes to render data on SciNet, and manipulate the results interactively on your own desktop. To use the paraview server on SciNet is much like using it locally, but there is an additional step in setting up a connection directly between your desktop and the compute nodes.

[[Image:Paraview.png|thumb|right|320px|The ParaView Client GUI]]

===Installing ParaView===

To use Paraview, you will have to have the client software installed on your system; you will need ParaView from [http://www.paraview.org/paraview/resources/software.html the Paraview website]. Binaries exist for Linux, Mac, and Windows systems. The client version must exactly match the version installed on the server, currently 3.12 or 3.14.1. The client version has all the functionality of the server, and can analyze data locally.

===SSH Forwarding For ParaView===

To interactively use the ParaView server on GPC, you will have to work some ssh magic to allow the client on your desktop to connect to the server through the scinet login nodes. The steps required are

* Have an SSH key that you can use to log into SciNet
* Submit an interactive job, with a shell on the head node that you'll be running the server on
* Start ssh forwarding
* Start paraview server
* Connecting client and server

====SSH Keys====

To be able to log into the compute nodes where ParaView will be running, you'll have to have an [[Ssh_keys | SSH key]] set up, as password authentication won't work. Our [[Ssh_keys | SSH Keys and SciNet]] page describes how to do this.

====Log into node====

The first thing to do is to go to the node from which you'll start the ParaView server. This is typically done by starting an interactive job on the GPC, perhaps on the [[Moab#debug | debug ]] queue or sandybridge [[GPC_Quickstart#Memory_Configuration | large memory]] nodes. Paraview can in principle make use of as many nodes as you throw at it. So one might begin jobs as below:

<pre>
qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy -I
</pre>
or
<pre>
qsub -l nodes=2:ppn=8,walltime=1:00:00 -q debug -I
</pre>

Once this job has started, you'll be placed in a shell on the head node of the job; typing `<pre>hostname</pre>' will tell you the name of the host, eg
<pre>
$ hostname
gpc-f148n089-ib0
</pre>
or
<pre>
$ hostname
gpc-f107n045-ib0
</pre>
you will need this hostname in the following steps.

====Start SSH port forwarding====

Once the ssh configuration is set, the port forwarding can be started with the command (on your local machine in a terminal window), using the local host name from above - here we'll take the example of gpc-f148n089-ib0:
<pre>
$ export gpcnode="gpc-f148n089-ib0"
$ ssh -N -L 20080:${gpcnode}:22 -L 20090:${gpcnode}:11111 login.scinet.utoronto.ca
</pre>
this command will not return anything until the forwarding is terminated, and will just look like it's sitting there. It doesn't start a remote shell or command (-N), but it will connect to login.scinet.utoronto.ca, and from there it will redirect your local (-L) port 20080 to <tt>${gpcnode}</tt> port 22, and similarly local port 20090 to <tt>${gpcnode}</tt> port 11111. We'll use the first for ssh'ing to the remote node (mainly for testing), and the second to conect the local paraview client to the remote paraview server.

To make sure the port forwarding is working correctly, in another window try sshing directly to the compute node from your desktop:
<pre>
$ ssh -p 20080 [your-scinet-username]@localhost
</pre>
and this should land you directly on the compute node. If it does not, then something is wrong with the ssh forwarding.

====Start Server====

Now that the tunnel is set up, on the compute node you can start the paraview server. To do this, you will have to have the following modules loaded:

<pre>
$ module load Xlibraries intel gcc python openmpi paraview
</pre>
(You can replace intelmpi with openmpi, and of course any module that is already loaded does have to be loaded again.)

Then start the paraview server with the intel mpirun as with any MPI job:
<pre>
$ mpirun -np [NP] pvserver --use-offscreen-rendering
</pre>
where NP is the number of processors; 16 processors per node on the largemem nodes, or 8 per node otherwise.

Once running, the ParaView server should output
<pre>
Listen on port: 11111
Waiting for client...
</pre>

====Connect Client and Server====

[[Image:Configure.png|thumb|right|320px|Configuring the client]]

Once the server is running, you can connect the client. Start the ParaView client on your desktop, and choose File->Connect. Click `Add Server', give the server a name (say, GPC), and give the port number 20090. The other values should be correct by default; host is <tt>localhost</tt>, and the server type is Client/Server. Click `Configure'.

On the next window, you'll be asked for a command to start up the server; select `Manual', and ok.

Once the server is selected, click `Connect'. On the compute node, the server should respond `Client connected'. In the client window, when you (for instance) select File->Open, you will be seeing the files on the GPC, rather than the local host.

From here, the [http://paraview.org/Wiki/ParaView ParaView Wiki] can give you instructions as to how to plot your data.

Using Paraview

2014-09-10T20:03:15Z

Ljdursi: /* SSH Forwarding For ParaView */

[http://www.paraview.org/ ParaView] is a powerful, parallel, client-server based visualization system that allows you to use SciNet's GPC nodes to render data on SciNet, and manipulate the results interactively on your own desktop. To use the paraview server on SciNet is much like using it locally, but there is an additional step in setting up a connection directly between your desktop and the compute nodes.

[[Image:Paraview.png|thumb|right|320px|The ParaView Client GUI]]

===Installing ParaView===

To use Paraview, you will have to have the client software installed on your system; you will need ParaView from [http://www.paraview.org/paraview/resources/software.html the Paraview website]. Binaries exist for Linux, Mac, and Windows systems. The client version must exactly match the version installed on the server, currently 3.12 or 3.14.1. The client version has all the functionality of the server, and can analyze data locally.

===SSH Forwarding For ParaView===

To interactively use the ParaView server on GPC, you will have to work some ssh magic to allow the client on your desktop to connect to the server through the scinet login nodes. The steps required are

* Have an SSH key that you can use to log into SciNet
* Submit an interactive job, with a shell on the head node that you'll be running the server on
* Start ssh forwarding
* Start paraview server
* Connecting client and server

====SSH Keys====

To be able to log into the compute nodes where ParaView will be running, you'll have to have an [[Ssh_keys | SSH key]] set up, as password authentication won't work. Our [[Ssh_keys | SSH Keys and SciNet]] page describes how to do this.

====Log into node====

The first thing to do is to go to the node from which you'll start the ParaView server. This is typically done by starting an interactive job on the GPC, perhaps on the [[Moab#debug | debug ]] queue or sandybridge [[GPC_Quickstart#Memory_Configuration | large memory]] nodes. Paraview can in principle make use of as many nodes as you throw at it. So one might begin jobs as below:

<pre>
qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy -I
</pre>
or
<pre>
qsub -l nodes=2:ppn=8,walltime=1:00:00 -q debug -I
</pre>

Once this job has started, you'll be placed in a shell on the head node of the job; typing `<pre>hostname</pre>' will tell you the name of the host, eg
<pre>
$ hostname
gpc-f148n089-ib0
</pre>
or
<pre>
$ hostname
gpc-f107n045-ib0
</pre>
you will need this hostname in the following steps.

====Start SSH port forwarding====

Once the ssh configuration is set, the port forwarding can be started with the command (on your local machine in a terminal window), using the local host name from above - here we'll take the example of gpc-f148n089-ib0:
<pre>
$ export gpcnode="gpc-f148n089-ib0"
$ ssh -N -L 20080:${gpcnode}:22 -L 20090:${gpcnode}:11111 login.scinet.utoronto.ca
</pre>
this command will not return anything until the forwarding is terminated, and will just look like it's sitting there. It doesn't start a remote shell or command (-N), but it will connect to login.scinet.utoronto.ca, and from there it will redirect your local (-L) port 20080 to <tt>${gpcnode}</tt> port 22, and similarly local port 20090 to <tt>${gpcnode}</tt> port 11111. We'll use the first for ssh'ing to the remote node (mainly for testing), and the second to conect the local paraview client to the remote paraview server.

To make sure the port forwarding is working correctly, in another window try sshing directly to the compute node from your desktop:
<pre>
$ ssh -p 20080 [your-scinet-username]@localhost
</pre>
and this should land you directly on the compute node. If it does not, then something is wrong with the ssh forwarding.

====Start Server====

Now that the tunnel is set up, on the compute node you can start the paraview server. To do this, you will have to have the following modules loaded:

<pre>
$ module load Xlibraries intel gcc python openmpi paraview
</pre>
(You can replace intelmpi with openmpi, and of course any module that is already loaded does have to be loaded again.)

Then start the paraview server with the intel mpirun as with any MPI job:
<pre>
$ mpirun -np [NP] pvserver --use-offscreen-rendering
</pre>
where NP is the number of processors; 16 processors per node on the largemem nodes, or 8 per node otherwise.

Once running, the ParaView server should output
<pre>
Listen on port: 11111
Waiting for client...
</pre>

====Connect Client and Server====

[[Image:Configure.png|thumb|right|320px|Configuring the client]]

Once the server is running, you can connect the client. Start the ParaView client on your desktop, and choose File->Connect. Click `Add Server', give the server a name (say, GPC), and give the port number 20090. The other values should be correct by default; host is <tt>localhost</tt>, and the server type is Client/Server. Click `Configure'.

On the next window, you'll be asked for a command to start up the server; select `Manual', and ok.

In future runs, you'll be able to re-use this server, even if the host is different, because the correct host will be set in your <tt>.ssh/config</tt>.

Once the server is selected, click `Connect'. On the compute node, the server should respond `Client connected'. In the client window, when you (for instance) select File->Open, you will be seeing the files on the GPC, rather than the local host.

From here, the [http://paraview.org/Wiki/ParaView ParaView Wiki] can give you instructions as to how to plot your data.

Using Paraview

2014-09-10T19:57:23Z

Ljdursi:

[http://www.paraview.org/ ParaView] is a powerful, parallel, client-server based visualization system that allows you to use SciNet's GPC nodes to render data on SciNet, and manipulate the results interactively on your own desktop. To use the paraview server on SciNet is much like using it locally, but there is an additional step in setting up a connection directly between your desktop and the compute nodes.

[[Image:Paraview.png|thumb|right|320px|The ParaView Client GUI]]

===Installing ParaView===

To use Paraview, you will have to have the client software installed on your system; you will need ParaView from [http://www.paraview.org/paraview/resources/software.html the Paraview website]. Binaries exist for Linux, Mac, and Windows systems. The client version must exactly match the version installed on the server, currently 3.12 or 3.14.1. The client version has all the functionality of the server, and can analyze data locally.

===SSH Forwarding For ParaView===

To interactively use the ParaView server on GPC, you will have to work some ssh magic to allow the client on your desktop to connect to the server through the scinet login nodes. The steps required are

* Have an SSH key that you can use to log into SciNet
* Log into the head node that you'll be using the server on
* (Mac or Linux): Edit your local <tt>~/.ssh/config</tt> to enable forwarding to that node
* Start ssh forwarding
* Start server
* Connecting client and server

====SSH Keys====

To be able to log into the compute nodes where ParaView will be running, you'll have to have an [[Ssh_keys | SSH key]] set up, as password authentication won't work. Our [[Ssh_keys | SSH Keys and SciNet]] page describes how to do this.

====Log into node====

The first thing to do is to go to the node from which you'll start the ParaView server. This is typically done by starting an interactive job on the GPC, perhaps on the [[Moab#debug | debug ]] queue or sandybridge [[GPC_Quickstart#Memory_Configuration | large memory]] nodes. Paraview can in principle make use of as many nodes as you throw at it. So one might begin jobs as below:

<pre>
qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy -I
</pre>
or
<pre>
qsub -l nodes=2:ppn=8,walltime=1:00:00 -q debug -I
</pre>

Once this job has started, you'll be placed in a shell on the head node of the job; typing `<pre>hostname</pre>' will tell you the name of the host, eg
<pre>
$ hostname
gpc-f148n089-ib0
</pre>
or
<pre>
$ hostname
gpc-f107n045-ib0
</pre>
you will need this hostname in the following steps.

====Start SSH port forwarding====

Once the ssh configuration is set, the port forwarding can be started with the command (on your local machine in a terminal window), using the local host name from above - here we'll take the example of gpc-f148n089-ib0:
<pre>
$ export gpcnode="gpc-f148n089-ib0"
$ ssh -N -L 20080:${gpcnode}:22 -L 20090:${gpcnode}:11111 login.scinet.utoronto.ca
</pre>
this command will not return anything until the forwarding is terminated, and will just look like it's sitting there. It doesn't start a remote shell or command (-N), but it will connect to login.scinet.utoronto.ca, and from there it will redirect your local (-L) port 20080 to <tt>${gpcnode}</tt> port 22, and similarly local port 20090 to <tt>${gpcnode}</tt> port 11111. We'll use the first for ssh'ing to the remote node (mainly for testing), and the second to conect the local paraview client to the remote paraview server.

To make sure the port forwarding is working correctly, in another window try sshing directly to the compute node from your desktop:
<pre>
$ ssh -p 20080 [your-scinet-username]@localhost
</pre>
and this should land you directly on the compute node. If it does not, then something is wrong with the ssh forwarding.

====Start Server====

Now that the tunnel is set up, on the compute node you can start the paraview server. To do this, you will have to have the following modules loaded:

<pre>
$ module load Xlibraries intel gcc python openmpi paraview
</pre>
(You can replace intelmpi with openmpi, and of course any module that is already loaded does have to be loaded again.)

Then start the paraview server with the intel mpirun as with any MPI job:
<pre>
$ mpirun -np [NP] pvserver --use-offscreen-rendering
</pre>
where NP is the number of processors; 16 processors per node on the largemem nodes, or 8 per node otherwise.

Once running, the ParaView server should output
<pre>
Listen on port: 11111
Waiting for client...
</pre>

====Connect Client and Server====

[[Image:Configure.png|thumb|right|320px|Configuring the client]]

Once the server is running, you can connect the client. Start the ParaView client on your desktop, and choose File->Connect. Click `Add Server', give the server a name (say, GPC), and give the port number 20090. The other values should be correct by default; host is <tt>localhost</tt>, and the server type is Client/Server. Click `Configure'.

On the next window, you'll be asked for a command to start up the server; select `Manual', and ok.

In future runs, you'll be able to re-use this server, even if the host is different, because the correct host will be set in your <tt>.ssh/config</tt>.

Once the server is selected, click `Connect'. On the compute node, the server should respond `Client connected'. In the client window, when you (for instance) select File->Open, you will be seeing the files on the GPC, rather than the local host.

From here, the [http://paraview.org/Wiki/ParaView ParaView Wiki] can give you instructions as to how to plot your data.

Using Paraview

2014-09-10T19:52:50Z

Ljdursi:

[http://www.paraview.org/ ParaView] is a powerful, parallel, client-server based visualization system that allows you to use SciNet's GPC nodes to render data on SciNet, and manipulate the results interactively on your own desktop. To use the paraview server on SciNet is much like using it locally, but there is an additional step in setting up a connection directly between your desktop and the compute nodes.

[[Image:Paraview.png|thumb|right|320px|The ParaView Client GUI]]

===Installing ParaView===

To use Paraview, you will have to have the client software installed on your system; you will need ParaView from [http://www.paraview.org/paraview/resources/software.html the Paraview website]. Binaries exist for Linux, Mac, and Windows systems. The client version must exactly match the version installed on the server, currently 3.12 or 3.14.1. The client version has all the functionality of the server, and can analyze data locally.

===SSH Forwarding For ParaView===

To interactively use the ParaView server on GPC, you will have to work some ssh magic to allow the client on your desktop to connect to the server through the scinet login nodes. The steps required are

* Have an SSH key that you can use to log into SciNet
* Log into the head node that you'll be using the server on
* (Mac or Linux): Edit your local <tt>~/.ssh/config</tt> to enable forwarding to that node
* Start ssh forwarding
* Start server
* Connecting client and server

====SSH Keys====

To be able to log into the compute nodes where ParaView will be running, you'll have to have an [[Ssh_keys | SSH key]] set up, as password authentication won't work. Our [[Ssh_keys | SSH Keys and SciNet]] page describes how to do this.

====Log into node====

The first thing to do is to go to the node from which you'll start the ParaView server. This is typically done by starting an interactive job on the GPC, perhaps on the [[Moab#debug | debug ]] queue or sandybridge [[GPC_Quickstart#Memory_Configuration | large memory]] nodes. Paraview can in principle make use of as many nodes as you throw at it. So one might begin jobs as below:

<pre>
qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy -I
</pre>
or
<pre>
qsub -l nodes=2:ppn=8,walltime=1:00:00 -q debug -I
</pre>

Once this job has started, you'll be placed in a shell on the head node of the job; typing `<pre>hostname</pre>' will tell you the name of the host, eg
<pre>
$ hostname
gpc-f148n089-ib0
</pre>
or
<pre>
$ hostname
gpc-f107n045-ib0
</pre>
you will need this hostname in the following steps; on your local machine, in the terminal, set a variable name <tt>gpcnode</tt> to the remote node name, eg
<pre>
export gpcnode=gpc-f148n089-ib0
</pre>

====Start SSH port forwarding====

Once the ssh configuration is set, the port forwarding can be started with the command (on your desktop, in the same terminal you set the gpcnode variable above:)

<pre>
$ ssh -N -L 20080:${gpcnode}:22 -L 20090:${gpcnode}:11111 login.scinet.utoronto.ca
</pre>
this command will not return anything until the forwarding is terminated, and will just look like it's sitting there. It doesn't start a remote shell or command (-N), but it will connect to login.scinet.utoronto.ca, and from there it will redirect your local (-L) port 20080 to <tt>${gpcnode}</tt> port 22, and similarly local port 20090 to <tt>${gpcnode}</tt> port 11111. We'll use the first for ssh'ing to the remote node (mainly for testing), and the second to conect the local paraview client to the remote paraview server.

To make sure the port forwarding is working correctly, in another window try sshing directly to the compute node from your desktop:
<pre>
$ ssh -p 20080 [your-scinet-username]@localhost
</pre>
and this should land you directly on the compute node. If it does not, then something is wrong with the ssh forwarding.

====Start Server====

Now that the tunnel is set up, on the compute node you can start the paraview server. To do this, you will have to have the following modules loaded:

<pre>
$ module load Xlibraries intel gcc python openmpi paraview
</pre>
(You can replace intelmpi with openmpi, and of course any module that is already loaded does have to be loaded again.)

Then start the paraview server with the intel mpirun as with any MPI job:
<pre>
$ mpirun -np [NP] pvserver --use-offscreen-rendering
</pre>
where NP is the number of processors; 16 processors per node on the largemem nodes, or 8 per node otherwise.

Once running, the ParaView server should output
<pre>
Listen on port: 11111
Waiting for client...
</pre>

====Connect Client and Server====

[[Image:Configure.png|thumb|right|320px|Configuring the client]]

Once the server is running, you can connect the client. Start the ParaView client on your desktop, and choose File->Connect. Click `Add Server', give the server a name (say, GPC), and give the port number 20090. The other values should be correct by default; host is <tt>localhost</tt>, and the server type is Client/Server. Click `Configure'.

On the next window, you'll be asked for a command to start up the server; select `Manual', and ok.

In future runs, you'll be able to re-use this server, even if the host is different, because the correct host will be set in your <tt>.ssh/config</tt>.

Once the server is selected, click `Connect'. On the compute node, the server should respond `Client connected'. In the client window, when you (for instance) select File->Open, you will be seeing the files on the GPC, rather than the local host.

From here, the [http://paraview.org/Wiki/ParaView ParaView Wiki] can give you instructions as to how to plot your data.

Using Paraview

2014-09-10T19:49:21Z

Ljdursi: /* Log into node */

[http://www.paraview.org/ ParaView] is a powerful, parallel, client-server based visualization system that allows you to use SciNet's GPC nodes to render data on SciNet, and manipulate the results interactively on your own desktop. To use the paraview server on SciNet is much like using it locally, but there is an additional step in setting up a connection directly between your desktop and the compute nodes.

[[Image:Paraview.png|thumb|right|320px|The ParaView Client GUI]]

===Installing ParaView===

To use Paraview, you will have to have the client software installed on your system; you will need ParaView from [http://www.paraview.org/paraview/resources/software.html the Paraview website]. Binaries exist for Linux, Mac, and Windows systems. The client version must exactly match the version installed on the server, currently 3.12 or 3.14.1. The client version has all the functionality of the server, and can analyze data locally.

===SSH Forwarding For ParaView===

To interactively use the ParaView server on GPC, you will have to work some ssh magic to allow the client on your desktop to connect to the server through the scinet login nodes. The steps required are

* Have an SSH key that you can use to log into SciNet
* Log into the head node that you'll be using the server on
* (Mac or Linux): Edit your local <tt>~/.ssh/config</tt> to enable forwarding to that node
* Start ssh forwarding
* Start server
* Connecting client and server

====SSH Keys====

To be able to log into the compute nodes where ParaView will be running, you'll have to have an [[Ssh_keys | SSH key]] set up, as password authentication won't work. Our [[Ssh_keys | SSH Keys and SciNet]] page describes how to do this.

====Log into node====

The first thing to do is to go to the node from which you'll start the ParaView server. This is typically done by starting an interactive job on the GPC, perhaps on the [[Moab#debug | debug ]] queue or sandybridge [[GPC_Quickstart#Memory_Configuration | large memory]] nodes. Paraview can in principle make use of as many nodes as you throw at it. So one might begin jobs as below:

<pre>
qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy -I
</pre>
or
<pre>
qsub -l nodes=2:ppn=8,walltime=1:00:00 -q debug -I
</pre>

Once this job has started, you'll be placed in a shell on the head node of the job; typing `<pre>hostname</pre>' will tell you the name of the host, eg
<pre>
$ hostname
gpc-f148n089-ib0
</pre>
or
<pre>
$ hostname
gpc-f107n045-ib0
</pre>
you will need this hostname in the following steps; on your local machine, in the terminal, set a variable name <tt>gpcnode</tt> to the remote node name, eg
<pre>
export gpcnode=gpc-f148n089-ib0
</pre>

====Edit ssh config (MacOS/Linux)====

You will now need to edit your ssh config to allow ssh forwarding so that you can seemingly connect directly to the compute node above. Add the following lines to your <tt>~/.ssh/config</tt> file in MacOS or Linux; windows users will have to consult their ssh client's documentation as to how to implement the forwarding:

<pre>
Host gpc_gw
HostName login.scinet.utoronto.ca
User [username]
LocalForward 20080 [hostname]:22
LocalForward 20090 [hostname]:11111

Host gpcnode
HostName localhost
HostKeyAlias gpcnode
User [username]
Port 20080
</pre>

Replace <tt>[username]</tt> with your username, and <tt>[hostname]</tt> with the name of the host from the previous step. This sets two ssh port forwards; one to port 11111 of the compute nodes, which is needed by ParaView; and one is just the usual SSH port 22, which can be used for testing. In future runs of the server, only the hostname in the first stanza needs to be changed.

====Edit ssh config (Windows, Cygwin)====

If you have Cygwin X installed on Windows, including the <tt>openssh</tt> package, take the following steps. First run in your Cygwin Bash Shell:
<pre>
$ ssh-user-config
</pre>
There is no need to create any of the keys, but this will create the ssh directory where you need to put the config file.

Next, go to the directory <tt>cygwin\home\[username]\.ssh\</tt>, where <tt>username</tt> is your computer login name. In this directory, create a file called <tt>config</tt> containing the code from the previous section, with the stanzas replaced by the appropriate host- and username. Make the file read-only.
The ssh port forwarding is now set up. Open a Cygwin Bash Shell and follow the rest of the instructions below.

====Start SSH port forwarding====

Once the ssh configuration is set, the port forwarding can be started with the command (on your desktop)

<pre>
$ ssh -N gpc_gw
</pre>
this command will not return anything until the forwarding is terminated, and will just look like it's sitting there. To make sure the port forwarding is working correctly, in another window try sshing directly to the compute node from your desktop:
<pre>
$ ssh gpcnode
</pre>
and this should land you directly on the compute node. If it does not, then something is wrong with the ssh forwarding.

====Start Server====

Now that the tunnel is set up, on the compute node you can start the paraview server. To do this, you will have to have the following modules loaded:

<pre>
$ module load Xlibraries intel gcc python openmpi paraview
</pre>
(You can replace intelmpi with openmpi, and of course any module that is already loaded does have to be loaded again.)

Then start the paraview server with the intel mpirun as with any MPI job:
<pre>
$ mpirun -np [NP] pvserver --use-offscreen-rendering
</pre>
where NP is the number of processors; 16 processors per node on the largemem nodes, or 8 per node otherwise.

Once running, the ParaView server should output
<pre>
Listen on port: 11111
Waiting for client...
</pre>

====Connect Client and Server====

[[Image:Configure.png|thumb|right|320px|Configuring the client]]

Once the server is running, you can connect the client. Start the ParaView client on your desktop, and choose File->Connect. Click `Add Server', give the server a name (say, GPC), and give the port number 20090. The other values should be correct by default; host is <tt>localhost</tt>, and the server type is Client/Server. Click `Configure'.

On the next window, you'll be asked for a command to start up the server; select `Manual', and ok.

In future runs, you'll be able to re-use this server, even if the host is different, because the correct host will be set in your <tt>.ssh/config</tt>.

Once the server is selected, click `Connect'. On the compute node, the server should respond `Client connected'. In the client window, when you (for instance) select File->Open, you will be seeing the files on the GPC, rather than the local host.

From here, the [http://paraview.org/Wiki/ParaView ParaView Wiki] can give you instructions as to how to plot your data.

Hadoop for HPCers

2014-09-03T17:28:55Z

Ljdursi:

=Overview=

This is a ~3 hour class that will introduce Hadoop to HPC users with a background in numerical simulation. We will walk through a brief overview of:
* The Hadoop File System (HDFS)
* Map Reduce
* Pig
* Spark

Most examples will be written in Python.

=VM Instructions=

This course will feature hands-on work with a 1-node Hadoop cluster running on your laptop. The VMs are created with [http://www.vagrantup.com Vagrant]. Before the course, ensure this is up and running:

* Install [https://www.virtualbox.org VirtualBox] on your laptop and start it. (Note! At time of writing, the newest version, 4.3.14, is broken on at least Mac and Windows; you'll want to install 4.3.12 from [https://www.virtualbox.org/wiki/Download_Old_Builds_4_3 "older builds"].)
* Under Settings or Preferences, go to Network, then Host-only networks, and add/create two host-only networks.
* Then download the virtual machine image you want to use:
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/GUI-VM.ova Full Size VM with GUI (require peak of ~8GB free disk space)]
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Text-VM.ova Smaller, Text-only (require peak of ~6GB free disk space)]
* "Import Appliance", and select the downloaded image; this will uncompress the image which will take some minutes.
* Start the new virtual machine.

If you get any warnings about shared folders not existing, that's fine.

The GUI VM will start up a console with a full desktop environment; you can open a terminal and begin working. For the text VM, you will have to login to the console; the username/password is vagrant/vagrant. For either machine, you can also ssh into the VM from your laptop from the terminal: <pre>ssh vagrant@192.168.33.10</pre> (or <pre>ssh -p 2222 vagrant@localhost</pre>) or to the laptop from the VM with <pre>ssh [username]@192.168.33.1</pre>.

(If that particular address pair doesn't work, from a window within the VM, type "ifconfig" to find a line like "inet addr: 192.168...." or "inet adde: 10. .."; that's the VMs IP address)

Then make sure everything is working:
* From a terminal, start up the hadoop cluster by typing <pre>~/bin/init.sh</pre> You may have to answer "yes" a few times to start up some servers.
* Go to one of the example directories by typing <pre>cd ~/examples/wordcount/streaming</pre>
* Then start the example by typing <pre>make</pre>

You've now run your (maybe) first Hadoop job!

If you'd like, you can also create the virtual machine image yourself by downloading [http://www.vagrantup.com Vagrant] and the Vagrantfile for the [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/Vagrantfile GUI] or [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Vagrantfile text] image and running "vagrant up". If you vagrant-up the GUI VM, you will have to "vagrant reload" after installation is completed to restart with all the software installed.

If you can't get the VM working for whatever reason, please contact us and we will make alternate arrangements.

= Updated Examples =

If you've downloaded the image before Wednesday morning, from within the VM you may want to download the updated examples from [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/examples.tgz https://support.scinet.utoronto.ca/~ljdursi/Hadoop/examples.tgz]

= Slides =

You can download [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/presentation.pdf the slides from here].

Hadoop for HPCers

2014-09-03T16:59:30Z

Ljdursi: /* VM Instructions */

Hadoop for HPCers

2014-09-03T13:08:52Z

Ljdursi: /* VM Instructions */

Hadoop for HPCers

2014-09-03T13:07:33Z

Ljdursi: /* VM Instructions */

=Overview=

This is a ~3 hour class that will introduce Hadoop to HPC users with a background in numerical simulation. We will walk through a brief overview of:
* The Hadoop File System (HDFS)
* Map Reduce
* Pig
* Spark

Most examples will be written in Python.

=VM Instructions=

This course will feature hands-on work with a 1-node Hadoop cluster running on your laptop. The VMs are created with [http://www.vagrantup.com Vagrant]. Before the course, ensure this is up and running:

* Install [https://www.virtualbox.org VirtualBox] on your laptop and start it. (Note! At time of writing, the newest version, 4.3.14, is broken on at least Mac and Windows; you'll want to install 4.3.12 from [https://www.virtualbox.org/wiki/Download_Old_Builds_4_3 "older builds"].)
* Under Settings or Preferences, go to Network, then Host-only networks, and add/create two host-only networks.
* Then download the virtual machine image you want to use:
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/GUI-VM.ova Full Size VM with GUI (require peak of ~8GB free disk space)]
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Text-VM.ova Smaller, Text-only (require peak of ~6GB free disk space)]
* "Import Appliance", and select the downloaded image; this will uncompress the image which will take some minutes.
* Start the new virtual machine.

The GUI VM will start up a console with a full desktop environment; you can open a terminal and begin working. For the text VM, you will have to login to the console; the username/password is vagrant/vagrant. For either machine, you can also ssh into the VM from your laptop from the terminal: <pre>ssh vagrant@192.168.33.10</pre> or to the laptop from the VM with <pre>ssh [username]@192.168.33.1</pre>

(If that particular address pair doesn't work, from a window within the VM, type "ifconfig | grep 192" to find a line like "inet addr: 192.168...."; that's the VMs IP address)

Then make sure everything is working:
* From a terminal, start up the hadoop cluster by typing <pre>~/bin/init.sh</pre> You may have to answer "yes" a few times to start up some servers.
* Go to one of the example directories by typing <pre>cd ~/examples/wordcount/streaming</pre>
* Then start the example by typing <pre>make</pre>

You've now run your (maybe) first Hadoop job!

If you'd like, you can also create the virtual machine image yourself by downloading [http://www.vagrantup.com Vagrant] and the Vagrantfile for the [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/Vagrantfile GUI] or [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Vagrantfile text] image and running "vagrant up". If you vagrant-up the GUI VM, you will have to "vagrant reload" after installation is completed to restart with all the software installed.

If you can't get the VM working for whatever reason, please contact us and we will make alternate arrangements.

Hadoop for HPCers

2014-09-03T13:05:58Z

Ljdursi: /* VM Instructions */

=Overview=

This is a ~3 hour class that will introduce Hadoop to HPC users with a background in numerical simulation. We will walk through a brief overview of:
* The Hadoop File System (HDFS)
* Map Reduce
* Pig
* Spark

Most examples will be written in Python.

=VM Instructions=

This course will feature hands-on work with a 1-node Hadoop cluster running on your laptop. The VMs are created with [http://www.vagrantup.com Vagrant]. Before the course, ensure this is up and running:

* Install [https://www.virtualbox.org VirtualBox] on your laptop and start it. (Note! At time of writing, the newest version, 4.3.14, is broken on at least Mac and Windows; you'll want to install 4.3.12 from "older builds".)
* Under Settings or Preferences, go to Network, then Host-only networks, and add/create two host-only networks.
* Then download the virtual machine image you want to use:
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/GUI-VM.ova Full Size VM with GUI (require peak of ~8GB free disk space)]
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Text-VM.ova Smaller, Text-only (require peak of ~6GB free disk space)]
* "Import Appliance", and select the downloaded image; this will uncompress the image which will take some minutes.
* Start the new virtual machine.

The GUI VM will start up a console with a full desktop environment; you can open a terminal and begin working. For the text VM, you will have to login to the console; the username/password is vagrant/vagrant. For either machine, you can also ssh into the VM from your laptop from the terminal: <pre>ssh vagrant@192.168.33.10</pre> or to the laptop from the VM with <pre>ssh [username]@192.168.33.1</pre>

(If that particular address pair doesn't work, from a window within the VM, type "ifconfig | grep 192" to find a line like "inet addr: 192.168...."; that's the VMs IP address)

Then make sure everything is working:
* From a terminal, start up the hadoop cluster by typing <pre>~/bin/init.sh</pre> You may have to answer "yes" a few times to start up some servers.
* Go to one of the example directories by typing <pre>cd ~/examples/wordcount/streaming</pre>
* Then start the example by typing <pre>make</pre>

You've now run your (maybe) first Hadoop job!

If you'd like, you can also create the virtual machine image yourself by downloading [http://www.vagrantup.com Vagrant] and the Vagrantfile for the [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/Vagrantfile GUI] or [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Vagrantfile text] image and running "vagrant up". If you vagrant-up the GUI VM, you will have to "vagrant reload" after installation is completed to restart with all the software installed.

If you can't get the VM working for whatever reason, please contact us and we will make alternate arrangements.

Hadoop for HPCers

2014-08-31T17:09:41Z

Ljdursi: /* VM Instructions */

=Overview=

This is a ~3 hour class that will introduce Hadoop to HPC users with a background in numerical simulation. We will walk through a brief overview of:
* The Hadoop File System (HDFS)
* Map Reduce
* Pig
* Spark

Most examples will be written in Python.

=VM Instructions=

This course will feature hands-on work with a 1-node Hadoop cluster running on your laptop. The VMs are created with [http://www.vagrantup.com Vagrant]. Before the course, ensure this is up and running:

* Install [https://www.virtualbox.org VirtualBox] on your laptop and start it.
* Under Settings or Preferences, go to Network, then Host-only networks, and add/create two host-only networks.
* Then download the virtual machine image you want to use:
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/GUI-VM.ova Full Size VM with GUI (require peak of ~8GB free disk space)]
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Text-VM.ova Smaller, Text-only (require peak of ~6GB free disk space)]
* "Import Appliance", and select the downloaded image; this will uncompress the image which will take some minutes.
* Start the new virtual machine.

The GUI VM will start up a console with a full desktop environment; you can open a terminal and begin working. For the text VM, you will have to login to the console; the username/password is vagrant/vagrant. For either machine, you can also ssh into the VM from your laptop from the terminal: <pre>ssh vagrant@192.168.33.10</pre> or to the laptop from the VM with <pre>ssh [username]@192.168.33.1</pre>

(If that particular address pair doesn't work, from a window within the VM, type "ifconfig | grep 192" to find a line like "inet addr: 192.168...."; that's the VMs IP address)

Then make sure everything is working:
* From a terminal, start up the hadoop cluster by typing <pre>~/bin/init.sh</pre> You may have to answer "yes" a few times to start up some servers.
* Go to one of the example directories by typing <pre>cd ~/examples/wordcount/streaming</pre>
* Then start the example by typing <pre>make</pre>

You've now run your (maybe) first Hadoop job!

If you'd like, you can also create the virtual machine image yourself by downloading [http://www.vagrantup.com Vagrant] and the Vagrantfile for the [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/Vagrantfile GUI] or [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Vagrantfile text] image and running "vagrant up". If you vagrant-up the GUI VM, you will have to "vagrant reload" after installation is completed to restart with all the software installed.

If you can't get the VM working for whatever reason, please contact us and we will make alternate arrangements.

Hadoop for HPCers

2014-08-29T18:23:06Z

Ljdursi: /* VM Instructions */

=Overview=

This is a ~3 hour class that will introduce Hadoop to HPC users with a background in numerical simulation. We will walk through a brief overview of:
* The Hadoop File System (HDFS)
* Map Reduce
* Pig
* Spark

Most examples will be written in Python.

=VM Instructions=

This course will feature hands-on work with a 1-node Hadoop cluster running on your laptop. The VMs are created with [http://www.vagrantup.com Vagrant]. Before the course, ensure this is up and running:

* Install [https://www.virtualbox.org VirtualBox] on your laptop and start it.
* Under Settings or Preferences, go to Network, then Host-only networks, and add/create two host-only networks.
* Then download the virtual machine image you want to use:
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/GUI-VM.ova Full Size VM with GUI (require peak of ~8GB free disk space)]
** [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Text-VM.ova Smaller, Text-only (require peak of ~6GB free disk space)]
* "Import Appliance", and select the downloaded image; this will uncompress the image which will take some minutes.
* Start the new virtual machine.

The GUI VM will start up a console with a full desktop environment; you can open a terminal and begin working. For the text VM, you will have to login to the console; the username/password is vagrant/vagrant. For either machine, you can also ssh into the VM from your laptop from the terminal: <pre>ssh vagrant@192.168.33.10</pre> or to the laptop from the VM with <pre>ssh [username]@192.168.33.1</pre>

(If that particular address pair doesn't work, from a window within the VM, type "ifconfig | grep 192" to find a line like "inet addr: 192.168...."; that's the VMs IP address)

Then make sure everything is working:
* From a terminal, start up the hadoop cluster by typing <pre>~/bin/init.sh</pre> You may have to answer "yes" a few times to start up some servers.
* Go to one of the example directories by typing <pre>cd ~/examples/wordcount/streaming</pre>
* Then start the example by typing <pre>make</pre>

You've now run your (maybe) first Hadoop job!

If you'd like, you can also create the virtual machine image yourself by downloading [http://www.vagrantup.com Vagrant] and the Vagrantfile for the [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/GUI/Vagrantfile GUI] or [https://support.scinet.utoronto.ca/~ljdursi/Hadoop/VMs/text/Vagrantfile text] image and running "vagrant up".

If you can't get the VM working for whatever reason, please contact us and we will make alternate arrangements.

Hadoop for HPCers

2014-08-29T14:06:38Z

Ljdursi:

Hadoop for HPCers

2014-08-29T12:32:56Z

Ljdursi:

Hadoop for HPCers

2014-08-29T12:20:57Z

Ljdursi:

Hadoop for HPCers

2014-08-29T12:18:44Z

Ljdursi: Created page with " =Overview= This is a ~3 hour class that will introduce Hadoop to HPC users with a background in numerical simulation. We will walk through a brief overview of: * The Hadoop..."

FAQ

2014-08-20T20:09:52Z

Ljdursi: Time to stop talking about 2012 allocations.

__TOC__

==The Basics==
===Whom do I contact for support?===

Whom do I contact if I have problems or questions about how to use the SciNet systems?

'''Answer:'''

E-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

In your email, please include the following information:

* your username on SciNet
* the cluster that your question pertains to (GPC or TCS; SciNet is not a cluster!),
* any relevant error messages
* the commands you typed before the errors occured
* the path to your code (if applicable)
* the location of the job scripts (if applicable)
* the directory from which it was submitted (if applicable)
* a description of what it is supposed to do (if applicable)
* if your problem is about connecting to SciNet, the type of computer you are connecting from.

Note that your password should never, never, never be to sent to us, even if your question is about your account.

Try to avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team!

===What does ''code scaling'' mean?===

'''Answer:'''

Please see [[Introduction_To_Performance#Parallel_Speedup|A Performance Primer]]

===What do you mean by ''throughput''?===

'''Answer:'''

Please see [[Introduction_To_Performance#Throughput|A Performance Primer]].

Here is a simple example:

Suppose you need to do 10 computations. Say each of these runs for
1 day on 8 cores, but they take "only" 18 hours on 16 cores. What is the
fastest way to get all 10 computations done - as 8-core jobs or as
16-core jobs? Let us assume you have 2 nodes at your disposal.
The answer, after some simple arithmetic, is that running your 10
jobs as 8-core jobs will take 5 days, whereas if you ran them
as 16-core jobs it would take 7.5 days. Take your own conclusions...

===I changed my .bashrc/.bash_profile and now nothing works===

The default startup scripts provided by SciNet, and guidelines for them, can be found [[Important_.bashrc_guidelines|here]]. Certain things - like sourcing <tt>/etc/profile</tt>
and <tt>/etc/bashrc</tt> are ''required'' for various SciNet routines to work!

If the situation is so bad that you cannot even log in, please send email [mailto:support@scinet.utoronto.ca support].

===Could I have my login shell changed to (t)csh?===

The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" <tt>#!/bin/tcsh</tt> at the top.

===How can I run Matlab / IDL / Gaussian / my favourite commercial software at SciNet?===

'''Answer:'''

Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).

If your research group requires a commercial package that you already have or are willing to buy licenses for, contact us at [mailto:support@scinet.utoronto.ca support@scinet] and we can work together to find out if it is feasible to implement the packages licensing arrangement on the SciNet clusters, and if so, what is the the best way to do it.

Note that it is important that you contact us before installing commercially licensed software on SciNet machines, even if you have a way to do it in your own directory without requiring sysadmin intervention. It puts us in a very awkward position if someone is found to be running unlicensed or invalidly licensed software on our systems, so we need to be aware of what is being installed where.

===Do you have a recommended ssh program that will allow scinet access from Windows machines?===

'''Answer:'''

The [[Ssh#SSH_for_Windows_Users | SSH for Windows users]] programs we recommend are:

* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools, including ssh and X, all wrapped up into one executable.
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] - this is a terminal for windows that connects via ssh. It is a quick install and will get you up and running quickly. To set up your passphrase protected ssh key with putty, see [http://the.earth.li/~sgtatham/putty/0.61/htmldoc/Chapter8.html#pubkey here].
* [http://www.cygwin.com/ CygWin] - this is a whole linux-like environment for windows, which also includes an X window server so that you can display remote windows on your desktop. Make sure you include the openssh and X window system in the installation for full functionality. This is recommended if you will be doing a lot of work on Linux machines, as it makes a very similar environment available on your computer. To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.
 To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.

===My ssh key does not work! WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ===

'''Answer:'''

[[Ssh_keys#Testing_Your_Key | Testing Your Key]]

* If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.
Please contact your system administrator.
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of
this message.
Offending key in /home/<user>/.ssh/known_hosts:3
RSA host key for login.scinet.utoronto.ca
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has
changed and you have requested
</pre>

* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be
restarted with the new passphrase ssh key.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.
</pre>

===Can't forward X: "Warning: No xauth data; using fake authentication data", or "X11 connection rejected because of wrong authentication."===

I used to be able to forward X11 windows from SciNet to my home machine, but now I'm getting these messages; what's wrong?

'''Answer:'''

This very likely means that ssh/xauth can't update your ${HOME}/.Xauthority file.

The simplest pssible reason for this is that you've filled your 10GB /home quota and so can't write anything to your home directory. Use

<pre>
$ module load extras
$ diskUsage
</pre>

to check to see how close you are to your disk usage on ${HOME}.

Alternately, this could mean your .Xauthority file has become broken/corrupted/confused some how, in which case you can delete that file, and when you next log in you'll get a similar warning message involving creating .Xauthority, but things should work.

===How come I can not login to TCS?===

'''Answer:'''

A SciNet account doesn't automatically entitle you to TCS access. At a minimum, TCS jobs need to run on at least 32 cores (64 preferred because of Simultaneous Multi Threading - [[TCS_Quickstart#Node_configuration|SMT]] - on these nodes) and need the large memory (4GB/core) and bandwidth on the system. Essentially you need to be able to explain why the work can't be done on the GPC.

===How can I reset the password for my Compute Canada account?===

'''Answer:'''

You can reset your password for your Compute Canada account here:

https://ccdb.computecanada.org/security/forgot

===How can I change or reset the password for my SciNet account?===

'''Answer:'''

To reset your password at SciNet please go to [https://portal.scinet.utoronto.ca/password_resets Password reset page].

If you know your old password and want to change it, that can be done here:

https://portal.scinet.utoronto.ca/change_password

===Why am I getting the error "Permission denied (publickey,gssapi-with-mic,password)"?===

This error can pop up in a variety of situations: when trying to log in, or when after a job has finished, when the error and output files fail to be copied (there are other possible reasons for this failure as well -- see [[FAQ#My_GPC_job_died.2C_telling_me_.60Copy_Stageout_Files_Failed.27|My GPC job died, telling me:Copy Stageout Files Failed]]).
In most cases, the "Permission denioed" error is caused by incorrect permission of the (hidden) .ssh directory. Ssh is used for logging in as well as for the copying of the standard error and output files after a job.

For security reasons,
the directory .ssh should only be writable and readable to you, but yours
has read permission for everybody, and thus it fails. You can change
this by
<pre>
chmod 700 ~/.ssh
</pre>
And to be sure, also do
<pre>
chmod 600 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub ~/authorized_keys
</pre>

===ERROR:102: Tcl command execution failed? when loading modules ===
Modules sometimes require other modules to be loaded first.
Module will let you know if you didn’t.
For example:
<pre>
$ module purge
$ module load python
python/2.6.2(11):ERROR:151: Module ’python/2.6.2’ depends on one of the module(s) ’gcc/4.4.0’
python/2.6.2(11):ERROR:102: Tcl command execution failed: prereq gcc/4.4.0
$ gpc-f103n084-$ module load gcc python
$
</pre>

==Compiling your Code==

===How can I get g77 to work?===

The fortran 77 compilers on the GPC are ifort and gfortran. We have dropped support for g77. This has been a conscious decision. g77 (and the associated library libg2c) were completely replaced six years ago (Apr 2005) by the gcc 4.x branch, and haven't undergone any updates at all, even bug fixes, for over five years.
If we would install g77 and libg2c, we would have to deal with the inevitable confusion caused when users accidentally link against the old, broken, wrong versions of the gcc libraries instead of the correct current versions.

If your code for some reason specifically requires five-plus-year-old libraries, availability, compatibility, and unfixed-known-bug problems are only going to get worse for you over time, and this might be as good an opportunity as any to address those issues.

''A note on porting to gfortran or ifort:''

While gfortran and ifort are rather compatible with g77, one
important difference is that by default, gfortran does not preserve
local variables between function calls, while g77 does. Preserved
local variables are for instance often used in implementations of quasi-random number
generators. Proper fortran requires to declare such variables as SAVE
but not all old code does this.
Luckily, you can change gfortran's default behavior with the flag
<tt>-fno-automatic</tt>. For ifort, the corresponding flag is <tt>-noautomatic</tt>.

===Where is libg2c.so?===

libg2c.so is part of the g77 compiler, for which we dropped support. See [[#How can I get g77 to work on the GPC?]] for our reasons.

===Autoparallelization does not work!===

I compiled my code with the <tt>-qsmp=omp,auto</tt> option, and then I specified that it should be run with 64 threads - with
export OMP_NUM_THREADS=64

However, when I check the load using <tt>llq1 -n</tt>, it shows a load on the node of 1.37. Why?

'''Answer:'''

Using the autoparallelization will only get you so far. In fact, it usually does not do too much. What is helpful is to run the compiler with the <tt>-qreport</tt> option, and then read the output listing carefully to see where the compiler thought it could parallelize, where it could not, and the reasons for this. Then you can go back to your code and carefully try to address each of the issues brought up by the compiler.
We ''emphasize'' that this is just a rough first guide, and that the compilers are still not magical! For more sophisticated approaches to parallelizing your code, email us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] to set up an appointment with one
of our technical analysts.

===How do I link against the Intel Math Kernel Library?===

If you need to link in the Intel Math Kernel Library (MKL) libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code.

'''''Note that this give the link line for the command line. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''''

'''''Note too that, unless the integer arguments you will be passing to the MKL libraries are actually 64-bit integers, rather than the normal int or INTEGER types, you want to specify 32-bit integers (lp64) .'''''

===Can the compilers on the login nodes be disabled to prevent accidentally using them?===

'''Answer:'''

You can accomplish this by modifying your .bashrc to not load the compiler modules. See [[Important .bashrc guidelines]].

==="relocation truncated to fit: R_X86_64_PC32": Huh?===

What does this mean, and why can't I compile this code?

'''Answer:'''

Welcome to the joys of the x86 architecture! You're probably having trouble building arrays larger than 2GB, individually or together. Generally, you have to try to use the medium or large x86 `memory model'. For the intel compilers, this is specified with the compile options

-mcmodel=medium -shared-intel

==="feupdateenv is not implemented and will always fail"===

How do I get rid of this and what does it mean?

'''Answer:'''
First note that, as ominous as it sounds, this is really just a warning, and has to do with the intel math library. You can ignore it (unless you really are trying to manually change the exception handlers for floating point exceptions such as divide by zero), or take the safe road and get rid off it by linking with the intel math functions library:<pre>-limf</pre>See also [[#How do I link against the Intel Math Kernel Library?]]

===Cannot find rdmacm library when compiling on GPC===

I get the following error building my code on GPC: "<tt>ld: cannot find -lrdmacm</tt>". Where can I find this library?

'''Answer:'''

This library is part of the MPI libraries; if your compiler is having problems picking it up, it probably means you are mistakenly trying to compile on the login nodes (scinet01..scinet04). The login nodes aren't part of the GPC; they are for logging into the data centre only. From there you must go to the GPC or TCS development nodes to do any real work.

=== Why do I get this error when I try to compile: "icpc: error #10001: could not find directory in which /usr/bin/g++41 resides" ?===

You are trying to compile on the login nodes. As described in the wiki ( https://support.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Login ), or in the users guide you would have received with your account, Scinet supports two main clusters, with very different architectures. Compilation must be done on the development nodes of the appropriate cluster (in this case, gpc01-04). Thus, log into gpc01, gpc02, gpc03, or gpc04, and compile from there.

==Testing your Code==

=== Can I run a something for a short time on the development nodes? ===

I am in the process of playing around with the mpi calls in my code to get it to work. I do a lot of tests and each of them takes a couple of seconds only. Can I do this on the development nodes?

'''Answer:'''

Yes, as long as it's very brief (a few minutes). People use the development nodes
for their work, and you don't want to bog it down for people, and testing a real
code can chew up a lot more resources than compiling, etc. The procedures differ
depending on what machine you're using.

==== TCS ====

On the TCS you can run small MPI jobs on the tcs02 node, which is meant for
development use. But even for this test run on one node, you'll need a host file --
a list of hosts (in this case, all tcs-f11n06, which is the `real' name of tcs02)
that the job will run on. Create a file called `hostfile' containing the following:

tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06

for a 4-task run. When you invoke "poe" or "mpirun", there are runtime
arguments that you specify pointing to this file. You can also specify it
in an environment variable MP_HOSTFILE, so, if your file is in your /scratch directory, say
${SCRATCH}/hostfile, then you would do

<pre>
export MP_HOSTFILE=${SCRATCH}/hostfile
</pre>

in your shell. You will also need to create a <tt>.rhosts</tt> file in your
home director, again listing <tt>tcs-f11n06</tt> so that <tt>poe</tt>
can start jobs. After that you can simply run your program. You can use
mpiexec:

<pre>
mpiexec -n 4 my_test_program
</pre>

adding <tt> -hostfile /path/to/my/hostfile</tt> if you did not set the environment
variable above. Alternatively, you can run it with the poe command (do a "man poe" for details), or even by
just directly running it. In this case the number of MPI processes will by default
be the number of entries in your hostfile.

==== GPC ====

On the GPC one can run short test jobs on the GPC [[GPC_Quickstart#Compile.2FDevel_Nodes | development nodes ]]<tt>gpc01</tt>..<tt>gpc04</tt>;
if they are single-node jobs (which they should be) they don't need a hostfile. Even better, though, is to request an [[ Moab#Interactive | interactive ]] job and run the tests either in regular batch queue or using a short high availability [[ Moab#debug | debug ]] queue that is reserved for this purpose.

=== How do I run a longer (but still shorter than an hour) test job quickly ? ===

'''Answer'''

On the GPC there is a high turnover short queue called [[ Moab#debug | debug ]] that is designed for
this purpose. You can use it by adding
<pre>
#PBS -q debug
</pre>
to your submission script.

==Running your jobs==

===My job can't write to /home===

My code works fine when I test on the development nodes, but when I submit a job, or even run interactively in the development queue on GPC, it fails. What's wrong?

'''Answer:'''

As [[Data_Management#Home_Disk_Space | discussed]] [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf elsewhere], <tt>/home</tt> is mounted read-only on the compute nodes; you can only write to <tt>/home</tt> from the login nodes and devel nodes. (The [[GPC_Quickstart#128Glargemem | largemem nodes]] on GPC, in this respect, are more like devel nodes than compute nodes). In general, to run jobs you can read from <tt>/home</tt> but you'll have to write to <tt>/scratch</tt> (or, if you were allocated space through the RAC process, on <tt>/project</tt>). More information on SciNet filesytems can be found on our [[Data_Management | Data Management]] page.

===Error Submitting My Job: qsub: Bad UID for job execution MSG=ruserok failed ===

I write up a submission script as in the examples, but when I attempt to submit the job, I get the above error. What's wrong?

'''Answer:'''

This error will occur if you try to submit a job from the login nodes. The login nodes are the gateway to all of SciNet's systems (GPC, TCS, P7, ARC), which have different hardware and queueing systems. To submit a job, you must log into a development node for the particular cluster you are submitting to and submit from there.

===OpenMP on the TCS===

How do I run an OpenMP job on the TCS?

'''Answer:'''

Please look at the [[TCS_Quickstart#Submission_Script_for_an_OpenMP_Job | TCS Quickstart ]] page.

===Can I can use hybrid codes consisting of MPI and openMP on the GPC?===

'''Answer:'''

Yes. Please look at the [[GPC_Quickstart#Hybrid_MPI.2FOpenMP_jobs | GPC Quickstart ]] page.

===How do I run serial jobs on GPC?===

'''Answer''':

So it should be said first that SciNet is a parallel computing resource,
and our priority will always be parallel jobs. Having said that, if
you can make efficient use of the resources using serial jobs and get
good science done, that's good too, and we're happy to help you.

The GPC nodes each have 8 processing cores, and making efficient use of these
nodes means using all eight cores. As a result, we'd like to have the
users take up whole nodes (eg, run multiples of 8 jobs) at a time.

It depends on the nature of your job what the best strategy is. Several approaches are presented on the [[User_Serial|serial run wiki page]].

===Why can't I request only a single cpu for my job on GPC?===

'''Answer''':

On GPC, computers are allocated by the node - that is, in chunks of 8 processors. If you want to run a job that requires only one processor, you need to bundle the jobs into groups of 8, so as to not be wasting the other 7 for 48 hours. See [[User_Serial|serial run wiki page]].

===How do I run serial jobs on TCS?===

'''Answer''': You don't.

===But in the queue I found a user who is running jobs on GPC, each of which is using only one processor, so why can't I?===

'''Answer''':

The pradat* and atlaspt* jobs, amongst others, are jobs of the ATLAS high energy physics project. That they are reported as single cpu jobs is an artifact of the moab scheduler. They are in fact being automatically bundled into 8-job bundles but have to run individually to be compatible with their international grid-based systems.

===How do I use the ramdisk on GPC?===

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) ${SCRATCH}. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

''More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].''

===How can I automatically resubmit a job?===

Commonly you may have a job that you know will take longer to run than what is
permissible in the queue. As long as your program contains [[Checkpoints|checkpoint]] or
restart capability, you can have one job automatically submit the next. In
the following example it is assumed that the program finishes before
the 48 hour limit and then resubmits itself by logging into one
of the development nodes.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example submission script for auto resubmission
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code

# RESUBMIT 10 TIMES HERE
num=$NUM
if [ $num -lt 10 ]; then
num=$(($num+1))
ssh gpc01 "cd $PBS_O_WORKDIR; qsub ./script_name.sh -v NUM=$num";
fi
</source>

<pre>
qsub script_name.sh -v
</pre>

You can alternatively use [[ Moab#Job_Dependencies | Job dependencies ]] through the queuing system which will not start one job until another job has completed.

If your job can't be made to automatically stop before the 48 hour queue window, but it does write out checkpoints, you can use the timeout command to stop the program while you still have time to resubmit; for instance

<source lang="bash">
timeout 2850m ./run_my_code argument1 argument2
</source>

will run the program for 47.5 hours (2850 minutes), and then send it SIGTERM to exit the program.

===How can I pass in arguments to my submission script?===

If you wish to make your scripts more generic you can use qsub's ability
to pass in environment variables to pass in arguments to your script.
The following example shows a case where an input and an output
file are passed in on the qsub line. Multiple variables can be
passed in using the qsub "-v" option and comma delimited.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example of passing in arguments
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code -f $INFILE -o $OUTFILE
</source>

<pre>
qsub script_name.sh -v INFILE=input.txt,OUTFILE=outfile.txt
</pre>

=== How can I run a job longer than 48 hours? ===

'''Answer:'''

The SciNet queues have a queue limit of 48 hours. This is pretty typical for systems of its size in Canada and elsewhere, and larger systems commonly have shorter limits. The limits are there to ensure that every user gets a fair share of the system (so that no one user ties up lots of nodes for a long time), and for safety (so that if one memory board in one node fails in the middle of a very long job, you haven't lost a months' worth of work).

Since many of us have simulations that require more than that much time, most widely-used scientific applications have "checkpoint-restart" functionality, where every so often the complete state of the calculation is stored as a checkpoint file, and one can restart a simulation from one of these. In fact, these restart files tend to be quite useful for a number of purposes.

If your job will take longer, you will have to submit your job in multiple parts, restarting from a checkpoint each time. In this way, one can run a simulation much longer than the queue limit. In fact, one can even write job scripts which automatically re-submit themselves until a run is completed, using [[FAQ#How_can_I_automatically_resubmit_a_job.3F | automatic resubmission. ]]

=== Why did showstart say it would take 3 hours for my job to start before, and now it says my job will start in 10 hours? ===

'''Answer:'''

Please look at the [[FAQ#How_do_priorities_work.2Fwhy_did_that_job_jump_ahead_of_mine_in_the_queue.3F | How do priorities work/why did that job jump ahead of mine in the queue? ]] page.

===How do priorities work/why did that job jump ahead of mine in the queue?===

'''Answer:'''

The [[Moab | queueing system]] used on SciNet machines is a [http://en.wikipedia.org/wiki/Priority_queue Priority Queue]. Jobs enter the queue at the back of the queue, and slowly make their way to the front as those ahead of them are run; but a job that enters the queue with a higher priority can `cut in line'.

The main factor which determines priority is whether or not the user (or their PI) has an [http://wiki.scinethpc.ca/wiki/index.php/Application_Process RAC allocation]. These are competitively allocated grants of computer time; there is a call for proposals towards the end of every calendar year. Users with an allocation have high priorities in an attempt to make sure that they can use the amount of computer time the committees granted them. Their priority decreases as they approach their allotted usage over the current window of time; by the time that they have exhausted that allotted usage, their priority is the same as users with no allocation (unallocated, or `default' users). Unallocated users have a fixed, low, priority.

This priority system is called `fairshare'; the scheduler attempts to make sure everyone has their fair share of the machines, where the share that's fair has been determined by the allocation committee. The fairshare window is a rolling window of two weeks; that is, any time you have a job in the queue, the fairshare calculation of its priority is given by how much of your allocation of the machine has been used in the last 14 days.

A particular allocation might have some fraction of GPC - say 4% of the machine (if the PI had been allocated 10 million CPU hours on GPC). The allocations have labels; (called `Resource Allocation Proposal Identifiers', or RAPIs) they look something like

abc-123-ab

where abc-123 is the PIs CCRI, and the suffix specifies which of the allocations granted to the PI is to be used. These can be specified on a job-by-job basis. On GPC, one adds the line
#PBS -A RAPI
to your script; on TCS, one uses
# @ account_no = RAPI
If the allocation to charge isn't specified, a default is used; each user has such a default, which can be changed at the same portal where one changes one's password,

https://portal.scinet.utoronto.ca/

A jobs priority is determined primarily by the fairshare priority of the allocation it is being charged to; the previous 14 days worth of use under that allocation is calculated and compared to the allocated fraction (here, 5%) of the machine over that window (here, 14 days). The fairshare priority is a decreasing function of the allocation left; if there is no allocation left (eg, jobs running under that allocation have already used 379,038 CPU hours in the past 14 days), the priority is the same as that of a user with no granted allocation. (This last part has been the topic of some debate; as the machine gets more utilized, it will probably be the case that we allow RAC users who have greatly overused their quota to have their priorities to drop below that of unallocated users, to give the unallocated users some chance to run on our increasingly crowded system; this would have no undue effect on our allocated users as they still would be able to use the amount of resources they had been allocated by the committees.) Note that all jobs charging the same allocation get the same fairshare priority.

There are other factors that go into calculating priority, but fairshare is the most significant. Other factors include
* amount of time waiting in queue (measured in units of the requested runtime). A waiting queue job gains priority as it sits in the queue to avoid job starvation.
* User adjustment of priorities ( See below ).

The major effect of these subdominant terms is to shuffle the order of jobs running under the same allocation.

===How do we manage job priorities within our research group?===

'''Answer:'''

Obviously, managing shared resources within a large group - whether it
is conference funding or CPU time - takes some doing.

It's important to note that the fairshare periods are intentionally kept
quite short - just two weeks long. So, for example, let us say that in your resource
allocation you have about 10% of the machine. Then for someone to use
up the whole two week amount of time in 2 days, they'd have to use 70%
of the machine in those two days - which is unlikely to happen by
accident. If that does happen,
those using the same allocation as the person who used 70% of the
machine over the two days will suffer by having much lower priority for
their jobs, but only for the next 12 days - and even then, if there are
idle cpus they'll still be able to compute.

There will be online tools for seeing how the allocation is being used,
and those people who are in charge in your group will be able to use
that information to manage the users, telling them to dial it down or
up. We know that managing a large research group is hard, and we want
to make sure we provide you the information you need to do your job
effectively.

One way for users within a group to manage their priorities within the group
is with [[Moab#Adjusting_Job_Priority | user-adjusted priorities]]; this is
described in more detail on the [[Moab | Scheduling System]] page.

=== How do I charge jobs to my RAC allocation? ===

'''Answer:'''

Please see the [[Moab#Accounting|accounting section of Moab page]].

=== How does one check the amount of used CPU-hours in a project, and how does one get statistics for each user in the project? ===

'''Answer:'''

This information is available on the scinet portal,https://portal.scinet.utoronto.ca, See also [[SciNet Usage Reports]].

==Monitoring jobs in the queue==

===Why hasn't my job started?===

'''Answer:'''

Use the moab command

<pre>
checkjob -v jobid
</pre>

and the last couple of lines should explain why a job hasn't started.

Please see [[Moab| Job Scheduling System (Moab) ]] for more detailed information

===How do I figure out when my job will run?===

'''Answer:'''

Please see [[Moab#Available_Resources| Job Scheduling System (Moab) ]]



===I submit my GPC job, and I get an email saying it was rejected===
'''Answer:'''

This happens because the job you've submitted breaks one of the rules of the queues and is rejected. An email
is sent with the JOBID, JOBNAME, and the reason it was rejected. The following is an example where a job
requests more than 48 hours and was rejected.

<pre>
PBS Job Id: 3462493.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'wclimit too high for class 'batch_ib' (345600 > 172800)'
</pre>

Jobs on the TCS or GPC may only run for 48 hours at a time; this restriction greatly increases responsiveness of the queue and queue throughput for all our users. If your computation requires longer than that, as many do, you will have to [[ Checkpoints | checkpoint ]] your job and restart it after each 48-hour queue window. You can manually re-submit jobs, or if you can have your job cleanly exit before the 48 hour window, there are ways to [[ FAQ#How_can_I_automatically_resubmit_a_job.3F | automatically resubmit jobs ]].

Other rejections return a more cryptic error saying "job violates class configuration" such as follows:
<pre>
PBS Job Id: 3462409.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'user required by class 'batch''
</pre>

The most common problems that result in this error are:

* '''Incorrect number of processors per node''': Jobs on the GPC are scheduled per-node not per-core and since each node has 8 processor cores (ppn=8) the smallest job allowed is one node with 8 cores (nodes=1:ppn=8). For serial jobs users must bundle or batch them together in groups of 8. See [[ FAQ#How_do_I_run_serial_jobs_on_GPC.3F | How do I run serial jobs on GPC? ]]
* '''No number of nodes specified''': Jobs submitted to the main queue must request a specific number of nodes, either in the submission script (with a line like <tt>#PBS -l nodes=2:ppn=8</tt>) or on the command line (eg, <tt>qsub -l nodes=2:ppn=8,walltime=5:00:00 script.pbs</tt>). Note that for the debug queue, you can get away without specifying a number of nodes and a default of one will be assigned; for both technical and policy reasons, we do not enforce such a default for the main ("batch") queue.
* '''There is a 15 minute walltime minimum''' on all queues except debug and if you set your walltime less than this, it will be rejected.

===Running checkjob on my job gives me messages about JobFail and rejected===

Running checkjob on my job gives me messages that suggest my job has failed, as below: what did I do wrong?

<pre>
AName: test
State: Idle
Creds: user:xxxxxx group:xxxxxxxx account:xxxxxxxx class:batch_ib qos:ibqos
WallTime: 00:00:00 of 8:00:00
BecameEligible: Wed Jul 23 10:39:27
SubmitTime: Wed Jul 23 10:38:22
(Time Queued Total: 00:01:47 Eligible: 00:01:05)

Total Requested Tasks: 8

Req[0] TaskCount: 8 Partition: ALL
Opsys: centos6computeA Arch: --- Features: ---

Notification Events: JobFail

IWD: /scratch/x/xxxxxxxx/xxxxxxx/xxxxxxx
Partition List: torque,DDR
Flags: RESTARTABLE
Attr: checkpoint
StartPriority: 76
rejected for Opsys - (null)
rejected for State - (null)
rejected for Reserved - (null)
NOTE: job req cannot run in partition torque (available procs do not meet requirements : 0 of 8 procs found)
idle procs: 793 feasible procs: 0

Node Rejection Summary: [Opsys: 117][State: 2895][Reserved: 19]

NOTE: job violates constraints for partition SANDY (partition SANDY not in job partition mask)

NOTE: job violates constraints for partition GRAVITY (partition GRAVITY not in job partition mask)

rejected for State - (null)
NOTE:
</pre>

'''Answer:'''

The output from check job is a little cryptic in places, and if you are wondering why your job hasn't started yet, you might think that "rejection" and "JobFail" suggest that there's something wrong. But the above message is actually normal; you can use the <tt>showstart</tt> command on your job to get a (preliminary, subject to change) estimate as to when the job will start, and you'll find that it is in fact scheduled to start up in the near future.

In the above message:

* `Notification Events: JobFail` just means that, if notifications are enabled, you'll get a message if the job fails;
* `job req cannot run in partition torque` just means that the job cannot run just yet (that's why it's queued);
* `job req cannot run in dynamic partition DDR now (insufficient procs available: 0 < 8)` says why: there aren't processors available; and
* `job violates constraints for partition SANDY/GRAVITY` just means that the job isn't eligable to run in those paritcular (small) sections of the cluster.

that is, the above output is the normal and expected (if somewhat cryptic) explanation as to why the job is waiting - nothing to worry about.

===How can I monitor my running jobs on TCS?===

How can I monitor the load of TCS jobs?

'''Answer:'''

You can get more information with the command
/xcat/tools/tcs-scripts/LL/jobState.sh
which I alias as:
alias llq1='/xcat/tools/tcs-scripts/LL/jobState.sh'
If you run "llq1 -n" you will see a listing of jobs together with a lot of information, including the load.

==Errors in running jobs==



=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in
/var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

=== My GPC job died, telling me `Copy Stageout Files Failed' ===

'''Answer:'''

When a job runs on GPC, the script's standard output and error are redirected to
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:
<pre>
An error has occurred processing your job, see below.
request to copy stageout files failed on node
</pre>
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process and those files just never got copied.

Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.



===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or
network configurations may exhibit this problem at higher or lower MPI
task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

<pre>
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288
</pre>

With Intel MPI, you should be able to do

<pre>
module load intelmpi/4.0.3.008
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode
</pre>

to the same end.

For more information see [[GPC MPI Versions]].

===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===

'''Answer:'''

To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.

==Data on SciNet disks==

===How do I find out my disk usage?===

'''Answer:'''

The standard unix/linux utilities for finding the amount of disk space used by a directory are very slow, and notoriously inefficient on the GPFS filesystems that we run on the SciNet systems. There are utilities that very quickly report your disk usage:

The <tt>'''diskUsage'''</tt> command, available with the 'extras' module on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time.
This information is only updated hourly!

More information about these filesystems is available at the [[Data_Management | Data_Management]].

===How do I transfer data to/from SciNet?===

'''Answer:'''

All incoming connections to SciNet go through relatively low-speed connections to the <tt>login.scinet</tt> gateways, so using scp to copy files the same way you ssh in is not an effective way to move lots of data. Better tools are described in our page on [[Data_Management#Data_Transfer | Data Transfer]].

===My group works with data files of size 1-2 GB. Is this too large to transfer by scp to login.scinet.utoronto.ca ?===

'''Answer:'''

Generally, occasion transfers of data less than 10GB is perfectly acceptible to so through the login nodes. See [[Data_Management#Data_Transfer | Data Transfer]].

===How can I check if I have files in /scratch that are scheduled for automatic deletion?===

'''Answer:'''

Please see [[Storage_Quickstart#Scratch_Disk_Purging_Policy | Storage At SciNet]]

===How to allow my supervisor to manage files for me using ACL-based commands?===

'''Answer:'''

Please see [[Data_Management#File.2FOwnership_Management_.28ACL.29 | File/Ownership Management]]

===Can we buy extra storage space on SciNet?===

'''Answer:'''
Yes, please see [[Data_Management#Buying_storage_space_on_GPFS_or_HPSS | Buying storage space on GPFS or HPSS ]] for more details.

===Can I transfer files between BGQ and HPSS?===

'''Answer:'''
Yes, please see [https://support.scinet.utoronto.ca/wiki/index.php/BGQ#Bridge_to_HPSS Bridge to HPSS ] for more details.

==Keep 'em Coming!==

===Next question, please===

Send your question to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]; we'll answer it asap!

FAQ

2014-08-20T15:45:38Z

Ljdursi: /* I submit my GPC job, and I get an email saying it was rejected */

__TOC__

==The Basics==
===Whom do I contact for support?===

Whom do I contact if I have problems or questions about how to use the SciNet systems?

'''Answer:'''

E-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

In your email, please include the following information:

* your username on SciNet
* the cluster that your question pertains to (GPC or TCS; SciNet is not a cluster!),
* any relevant error messages
* the commands you typed before the errors occured
* the path to your code (if applicable)
* the location of the job scripts (if applicable)
* the directory from which it was submitted (if applicable)
* a description of what it is supposed to do (if applicable)
* if your problem is about connecting to SciNet, the type of computer you are connecting from.

Note that your password should never, never, never be to sent to us, even if your question is about your account.

Try to avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team!

===What does ''code scaling'' mean?===

'''Answer:'''

Please see [[Introduction_To_Performance#Parallel_Speedup|A Performance Primer]]

===What do you mean by ''throughput''?===

'''Answer:'''

Please see [[Introduction_To_Performance#Throughput|A Performance Primer]].

Here is a simple example:

Suppose you need to do 10 computations. Say each of these runs for
1 day on 8 cores, but they take "only" 18 hours on 16 cores. What is the
fastest way to get all 10 computations done - as 8-core jobs or as
16-core jobs? Let us assume you have 2 nodes at your disposal.
The answer, after some simple arithmetic, is that running your 10
jobs as 8-core jobs will take 5 days, whereas if you ran them
as 16-core jobs it would take 7.5 days. Take your own conclusions...

===I changed my .bashrc/.bash_profile and now nothing works===

The default startup scripts provided by SciNet, and guidelines for them, can be found [[Important_.bashrc_guidelines|here]]. Certain things - like sourcing <tt>/etc/profile</tt>
and <tt>/etc/bashrc</tt> are ''required'' for various SciNet routines to work!

If the situation is so bad that you cannot even log in, please send email [mailto:support@scinet.utoronto.ca support].

===Could I have my login shell changed to (t)csh?===

The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" <tt>#!/bin/tcsh</tt> at the top.

===How can I run Matlab / IDL / Gaussian / my favourite commercial software at SciNet?===

'''Answer:'''

Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).

If your research group requires a commercial package that you already have or are willing to buy licenses for, contact us at [mailto:support@scinet.utoronto.ca support@scinet] and we can work together to find out if it is feasible to implement the packages licensing arrangement on the SciNet clusters, and if so, what is the the best way to do it.

Note that it is important that you contact us before installing commercially licensed software on SciNet machines, even if you have a way to do it in your own directory without requiring sysadmin intervention. It puts us in a very awkward position if someone is found to be running unlicensed or invalidly licensed software on our systems, so we need to be aware of what is being installed where.

===Do you have a recommended ssh program that will allow scinet access from Windows machines?===

'''Answer:'''

The [[Ssh#SSH_for_Windows_Users | SSH for Windows users]] programs we recommend are:

* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools, including ssh and X, all wrapped up into one executable.
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] - this is a terminal for windows that connects via ssh. It is a quick install and will get you up and running quickly. To set up your passphrase protected ssh key with putty, see [http://the.earth.li/~sgtatham/putty/0.61/htmldoc/Chapter8.html#pubkey here].
* [http://www.cygwin.com/ CygWin] - this is a whole linux-like environment for windows, which also includes an X window server so that you can display remote windows on your desktop. Make sure you include the openssh and X window system in the installation for full functionality. This is recommended if you will be doing a lot of work on Linux machines, as it makes a very similar environment available on your computer. To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.
 To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.

===My ssh key does not work! WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ===

'''Answer:'''

[[Ssh_keys#Testing_Your_Key | Testing Your Key]]

* If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.
Please contact your system administrator.
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of
this message.
Offending key in /home/<user>/.ssh/known_hosts:3
RSA host key for login.scinet.utoronto.ca
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has
changed and you have requested
</pre>

* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be
restarted with the new passphrase ssh key.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.
</pre>

===Can't forward X: "Warning: No xauth data; using fake authentication data", or "X11 connection rejected because of wrong authentication."===

I used to be able to forward X11 windows from SciNet to my home machine, but now I'm getting these messages; what's wrong?

'''Answer:'''

This very likely means that ssh/xauth can't update your ${HOME}/.Xauthority file.

The simplest pssible reason for this is that you've filled your 10GB /home quota and so can't write anything to your home directory. Use

<pre>
$ module load extras
$ diskUsage
</pre>

to check to see how close you are to your disk usage on ${HOME}.

Alternately, this could mean your .Xauthority file has become broken/corrupted/confused some how, in which case you can delete that file, and when you next log in you'll get a similar warning message involving creating .Xauthority, but things should work.

===How come I can not login to TCS?===

'''Answer:'''

A SciNet account doesn't automatically entitle you to TCS access. At a minimum, TCS jobs need to run on at least 32 cores (64 preferred because of Simultaneous Multi Threading - [[TCS_Quickstart#Node_configuration|SMT]] - on these nodes) and need the large memory (4GB/core) and bandwidth on the system. Essentially you need to be able to explain why the work can't be done on the GPC.

===How can I reset the password for my Compute Canada account?===

'''Answer:'''

You can reset your password for your Compute Canada account here:

https://ccdb.computecanada.org/security/forgot

===How can I change or reset the password for my SciNet account?===

'''Answer:'''

To reset your password at SciNet please go to [https://portal.scinet.utoronto.ca/password_resets Password reset page].

If you know your old password and want to change it, that can be done here:

https://portal.scinet.utoronto.ca/change_password

===Why am I getting the error "Permission denied (publickey,gssapi-with-mic,password)"?===

This error can pop up in a variety of situations: when trying to log in, or when after a job has finished, when the error and output files fail to be copied (there are other possible reasons for this failure as well -- see [[FAQ#My_GPC_job_died.2C_telling_me_.60Copy_Stageout_Files_Failed.27|My GPC job died, telling me:Copy Stageout Files Failed]]).
In most cases, the "Permission denioed" error is caused by incorrect permission of the (hidden) .ssh directory. Ssh is used for logging in as well as for the copying of the standard error and output files after a job.

For security reasons,
the directory .ssh should only be writable and readable to you, but yours
has read permission for everybody, and thus it fails. You can change
this by
<pre>
chmod 700 ~/.ssh
</pre>
And to be sure, also do
<pre>
chmod 600 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub ~/authorized_keys
</pre>

===ERROR:102: Tcl command execution failed? when loading modules ===
Modules sometimes require other modules to be loaded first.
Module will let you know if you didn’t.
For example:
<pre>
$ module purge
$ module load python
python/2.6.2(11):ERROR:151: Module ’python/2.6.2’ depends on one of the module(s) ’gcc/4.4.0’
python/2.6.2(11):ERROR:102: Tcl command execution failed: prereq gcc/4.4.0
$ gpc-f103n084-$ module load gcc python
$
</pre>

==Compiling your Code==

===How can I get g77 to work?===

The fortran 77 compilers on the GPC are ifort and gfortran. We have dropped support for g77. This has been a conscious decision. g77 (and the associated library libg2c) were completely replaced six years ago (Apr 2005) by the gcc 4.x branch, and haven't undergone any updates at all, even bug fixes, for over five years.
If we would install g77 and libg2c, we would have to deal with the inevitable confusion caused when users accidentally link against the old, broken, wrong versions of the gcc libraries instead of the correct current versions.

If your code for some reason specifically requires five-plus-year-old libraries, availability, compatibility, and unfixed-known-bug problems are only going to get worse for you over time, and this might be as good an opportunity as any to address those issues.

''A note on porting to gfortran or ifort:''

While gfortran and ifort are rather compatible with g77, one
important difference is that by default, gfortran does not preserve
local variables between function calls, while g77 does. Preserved
local variables are for instance often used in implementations of quasi-random number
generators. Proper fortran requires to declare such variables as SAVE
but not all old code does this.
Luckily, you can change gfortran's default behavior with the flag
<tt>-fno-automatic</tt>. For ifort, the corresponding flag is <tt>-noautomatic</tt>.

===Where is libg2c.so?===

libg2c.so is part of the g77 compiler, for which we dropped support. See [[#How can I get g77 to work on the GPC?]] for our reasons.

===Autoparallelization does not work!===

I compiled my code with the <tt>-qsmp=omp,auto</tt> option, and then I specified that it should be run with 64 threads - with
export OMP_NUM_THREADS=64

However, when I check the load using <tt>llq1 -n</tt>, it shows a load on the node of 1.37. Why?

'''Answer:'''

Using the autoparallelization will only get you so far. In fact, it usually does not do too much. What is helpful is to run the compiler with the <tt>-qreport</tt> option, and then read the output listing carefully to see where the compiler thought it could parallelize, where it could not, and the reasons for this. Then you can go back to your code and carefully try to address each of the issues brought up by the compiler.
We ''emphasize'' that this is just a rough first guide, and that the compilers are still not magical! For more sophisticated approaches to parallelizing your code, email us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] to set up an appointment with one
of our technical analysts.

===How do I link against the Intel Math Kernel Library?===

If you need to link in the Intel Math Kernel Library (MKL) libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code.

'''''Note that this give the link line for the command line. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''''

'''''Note too that, unless the integer arguments you will be passing to the MKL libraries are actually 64-bit integers, rather than the normal int or INTEGER types, you want to specify 32-bit integers (lp64) .'''''

===Can the compilers on the login nodes be disabled to prevent accidentally using them?===

'''Answer:'''

You can accomplish this by modifying your .bashrc to not load the compiler modules. See [[Important .bashrc guidelines]].

==="relocation truncated to fit: R_X86_64_PC32": Huh?===

What does this mean, and why can't I compile this code?

'''Answer:'''

Welcome to the joys of the x86 architecture! You're probably having trouble building arrays larger than 2GB, individually or together. Generally, you have to try to use the medium or large x86 `memory model'. For the intel compilers, this is specified with the compile options

-mcmodel=medium -shared-intel

==="feupdateenv is not implemented and will always fail"===

How do I get rid of this and what does it mean?

'''Answer:'''
First note that, as ominous as it sounds, this is really just a warning, and has to do with the intel math library. You can ignore it (unless you really are trying to manually change the exception handlers for floating point exceptions such as divide by zero), or take the safe road and get rid off it by linking with the intel math functions library:<pre>-limf</pre>See also [[#How do I link against the Intel Math Kernel Library?]]

===Cannot find rdmacm library when compiling on GPC===

I get the following error building my code on GPC: "<tt>ld: cannot find -lrdmacm</tt>". Where can I find this library?

'''Answer:'''

This library is part of the MPI libraries; if your compiler is having problems picking it up, it probably means you are mistakenly trying to compile on the login nodes (scinet01..scinet04). The login nodes aren't part of the GPC; they are for logging into the data centre only. From there you must go to the GPC or TCS development nodes to do any real work.

=== Why do I get this error when I try to compile: "icpc: error #10001: could not find directory in which /usr/bin/g++41 resides" ?===

You are trying to compile on the login nodes. As described in the wiki ( https://support.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Login ), or in the users guide you would have received with your account, Scinet supports two main clusters, with very different architectures. Compilation must be done on the development nodes of the appropriate cluster (in this case, gpc01-04). Thus, log into gpc01, gpc02, gpc03, or gpc04, and compile from there.

==Testing your Code==

=== Can I run a something for a short time on the development nodes? ===

I am in the process of playing around with the mpi calls in my code to get it to work. I do a lot of tests and each of them takes a couple of seconds only. Can I do this on the development nodes?

'''Answer:'''

Yes, as long as it's very brief (a few minutes). People use the development nodes
for their work, and you don't want to bog it down for people, and testing a real
code can chew up a lot more resources than compiling, etc. The procedures differ
depending on what machine you're using.

==== TCS ====

On the TCS you can run small MPI jobs on the tcs02 node, which is meant for
development use. But even for this test run on one node, you'll need a host file --
a list of hosts (in this case, all tcs-f11n06, which is the `real' name of tcs02)
that the job will run on. Create a file called `hostfile' containing the following:

tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06

for a 4-task run. When you invoke "poe" or "mpirun", there are runtime
arguments that you specify pointing to this file. You can also specify it
in an environment variable MP_HOSTFILE, so, if your file is in your /scratch directory, say
${SCRATCH}/hostfile, then you would do

<pre>
export MP_HOSTFILE=${SCRATCH}/hostfile
</pre>

in your shell. You will also need to create a <tt>.rhosts</tt> file in your
home director, again listing <tt>tcs-f11n06</tt> so that <tt>poe</tt>
can start jobs. After that you can simply run your program. You can use
mpiexec:

<pre>
mpiexec -n 4 my_test_program
</pre>

adding <tt> -hostfile /path/to/my/hostfile</tt> if you did not set the environment
variable above. Alternatively, you can run it with the poe command (do a "man poe" for details), or even by
just directly running it. In this case the number of MPI processes will by default
be the number of entries in your hostfile.

==== GPC ====

On the GPC one can run short test jobs on the GPC [[GPC_Quickstart#Compile.2FDevel_Nodes | development nodes ]]<tt>gpc01</tt>..<tt>gpc04</tt>;
if they are single-node jobs (which they should be) they don't need a hostfile. Even better, though, is to request an [[ Moab#Interactive | interactive ]] job and run the tests either in regular batch queue or using a short high availability [[ Moab#debug | debug ]] queue that is reserved for this purpose.

=== How do I run a longer (but still shorter than an hour) test job quickly ? ===

'''Answer'''

On the GPC there is a high turnover short queue called [[ Moab#debug | debug ]] that is designed for
this purpose. You can use it by adding
<pre>
#PBS -q debug
</pre>
to your submission script.

==Running your jobs==

===My job can't write to /home===

My code works fine when I test on the development nodes, but when I submit a job, or even run interactively in the development queue on GPC, it fails. What's wrong?

'''Answer:'''

As [[Data_Management#Home_Disk_Space | discussed]] [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf elsewhere], <tt>/home</tt> is mounted read-only on the compute nodes; you can only write to <tt>/home</tt> from the login nodes and devel nodes. (The [[GPC_Quickstart#128Glargemem | largemem nodes]] on GPC, in this respect, are more like devel nodes than compute nodes). In general, to run jobs you can read from <tt>/home</tt> but you'll have to write to <tt>/scratch</tt> (or, if you were allocated space through the RAC process, on <tt>/project</tt>). More information on SciNet filesytems can be found on our [[Data_Management | Data Management]] page.

===Error Submitting My Job: qsub: Bad UID for job execution MSG=ruserok failed ===

I write up a submission script as in the examples, but when I attempt to submit the job, I get the above error. What's wrong?

'''Answer:'''

This error will occur if you try to submit a job from the login nodes. The login nodes are the gateway to all of SciNet's systems (GPC, TCS, P7, ARC), which have different hardware and queueing systems. To submit a job, you must log into a development node for the particular cluster you are submitting to and submit from there.

===OpenMP on the TCS===

How do I run an OpenMP job on the TCS?

'''Answer:'''

Please look at the [[TCS_Quickstart#Submission_Script_for_an_OpenMP_Job | TCS Quickstart ]] page.

===Can I can use hybrid codes consisting of MPI and openMP on the GPC?===

'''Answer:'''

Yes. Please look at the [[GPC_Quickstart#Hybrid_MPI.2FOpenMP_jobs | GPC Quickstart ]] page.

===How do I run serial jobs on GPC?===

'''Answer''':

So it should be said first that SciNet is a parallel computing resource,
and our priority will always be parallel jobs. Having said that, if
you can make efficient use of the resources using serial jobs and get
good science done, that's good too, and we're happy to help you.

The GPC nodes each have 8 processing cores, and making efficient use of these
nodes means using all eight cores. As a result, we'd like to have the
users take up whole nodes (eg, run multiples of 8 jobs) at a time.

It depends on the nature of your job what the best strategy is. Several approaches are presented on the [[User_Serial|serial run wiki page]].

===Why can't I request only a single cpu for my job on GPC?===

'''Answer''':

On GPC, computers are allocated by the node - that is, in chunks of 8 processors. If you want to run a job that requires only one processor, you need to bundle the jobs into groups of 8, so as to not be wasting the other 7 for 48 hours. See [[User_Serial|serial run wiki page]].

===How do I run serial jobs on TCS?===

'''Answer''': You don't.

===But in the queue I found a user who is running jobs on GPC, each of which is using only one processor, so why can't I?===

'''Answer''':

The pradat* and atlaspt* jobs, amongst others, are jobs of the ATLAS high energy physics project. That they are reported as single cpu jobs is an artifact of the moab scheduler. They are in fact being automatically bundled into 8-job bundles but have to run individually to be compatible with their international grid-based systems.

===How do I use the ramdisk on GPC?===

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) ${SCRATCH}. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

''More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].''

===How can I automatically resubmit a job?===

Commonly you may have a job that you know will take longer to run than what is
permissible in the queue. As long as your program contains [[Checkpoints|checkpoint]] or
restart capability, you can have one job automatically submit the next. In
the following example it is assumed that the program finishes before
the 48 hour limit and then resubmits itself by logging into one
of the development nodes.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example submission script for auto resubmission
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code

# RESUBMIT 10 TIMES HERE
num=$NUM
if [ $num -lt 10 ]; then
num=$(($num+1))
ssh gpc01 "cd $PBS_O_WORKDIR; qsub ./script_name.sh -v NUM=$num";
fi
</source>

<pre>
qsub script_name.sh -v
</pre>

You can alternatively use [[ Moab#Job_Dependencies | Job dependencies ]] through the queuing system which will not start one job until another job has completed.

If your job can't be made to automatically stop before the 48 hour queue window, but it does write out checkpoints, you can use the timeout command to stop the program while you still have time to resubmit; for instance

<source lang="bash">
timeout 2850m ./run_my_code argument1 argument2
</source>

will run the program for 47.5 hours (2850 minutes), and then send it SIGTERM to exit the program.

===How can I pass in arguments to my submission script?===

If you wish to make your scripts more generic you can use qsub's ability
to pass in environment variables to pass in arguments to your script.
The following example shows a case where an input and an output
file are passed in on the qsub line. Multiple variables can be
passed in using the qsub "-v" option and comma delimited.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example of passing in arguments
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code -f $INFILE -o $OUTFILE
</source>

<pre>
qsub script_name.sh -v INFILE=input.txt,OUTFILE=outfile.txt
</pre>

=== How can I run a job longer than 48 hours? ===

'''Answer:'''

The SciNet queues have a queue limit of 48 hours. This is pretty typical for systems of its size in Canada and elsewhere, and larger systems commonly have shorter limits. The limits are there to ensure that every user gets a fair share of the system (so that no one user ties up lots of nodes for a long time), and for safety (so that if one memory board in one node fails in the middle of a very long job, you haven't lost a months' worth of work).

Since many of us have simulations that require more than that much time, most widely-used scientific applications have "checkpoint-restart" functionality, where every so often the complete state of the calculation is stored as a checkpoint file, and one can restart a simulation from one of these. In fact, these restart files tend to be quite useful for a number of purposes.

If your job will take longer, you will have to submit your job in multiple parts, restarting from a checkpoint each time. In this way, one can run a simulation much longer than the queue limit. In fact, one can even write job scripts which automatically re-submit themselves until a run is completed, using [[FAQ#How_can_I_automatically_resubmit_a_job.3F | automatic resubmission. ]]

=== Why did showstart say it would take 3 hours for my job to start before, and now it says my job will start in 10 hours? ===

'''Answer:'''

Please look at the [[FAQ#How_do_priorities_work.2Fwhy_did_that_job_jump_ahead_of_mine_in_the_queue.3F | How do priorities work/why did that job jump ahead of mine in the queue? ]] page.

===How do priorities work/why did that job jump ahead of mine in the queue?===

'''Answer:'''

The [[Moab | queueing system]] used on SciNet machines is a [http://en.wikipedia.org/wiki/Priority_queue Priority Queue]. Jobs enter the queue at the back of the queue, and slowly make their way to the front as those ahead of them are run; but a job that enters the queue with a higher priority can `cut in line'.

The main factor which determines priority is whether or not the user (or their PI) has an [http://wiki.scinethpc.ca/wiki/index.php/Application_Process RAC allocation]. These are competitively allocated grants of computer time; there is a call for proposals towards the end of every calendar year. Users with an allocation have high priorities in an attempt to make sure that they can use the amount of computer time the committees granted them. Their priority decreases as they approach their allotted usage over the current window of time; by the time that they have exhausted that allotted usage, their priority is the same as users with no allocation (unallocated, or `default' users). Unallocated users have a fixed, low, priority.

This priority system is called `fairshare'; the scheduler attempts to make sure everyone has their fair share of the machines, where the share that's fair has been determined by the allocation committee. The fairshare window is a rolling window of two weeks; that is, any time you have a job in the queue, the fairshare calculation of its priority is given by how much of your allocation of the machine has been used in the last 14 days.

A particular allocation might have some fraction of GPC - say 4% of the machine (if the PI had been allocated 10 million CPU hours on GPC). The allocations have labels; (called `Resource Allocation Proposal Identifiers', or RAPIs) they look something like

abc-123-ab

where abc-123 is the PIs CCRI, and the suffix specifies which of the allocations granted to the PI is to be used. These can be specified on a job-by-job basis. On GPC, one adds the line
#PBS -A RAPI
to your script; on TCS, one uses
# @ account_no = RAPI
If the allocation to charge isn't specified, a default is used; each user has such a default, which can be changed at the same portal where one changes one's password,

https://portal.scinet.utoronto.ca/

A jobs priority is determined primarily by the fairshare priority of the allocation it is being charged to; the previous 14 days worth of use under that allocation is calculated and compared to the allocated fraction (here, 5%) of the machine over that window (here, 14 days). The fairshare priority is a decreasing function of the allocation left; if there is no allocation left (eg, jobs running under that allocation have already used 379,038 CPU hours in the past 14 days), the priority is the same as that of a user with no granted allocation. (This last part has been the topic of some debate; as the machine gets more utilized, it will probably be the case that we allow RAC users who have greatly overused their quota to have their priorities to drop below that of unallocated users, to give the unallocated users some chance to run on our increasingly crowded system; this would have no undue effect on our allocated users as they still would be able to use the amount of resources they had been allocated by the committees.) Note that all jobs charging the same allocation get the same fairshare priority.

There are other factors that go into calculating priority, but fairshare is the most significant. Other factors include
* amount of time waiting in queue (measured in units of the requested runtime). A waiting queue job gains priority as it sits in the queue to avoid job starvation.
* User adjustment of priorities ( See below ).

The major effect of these subdominant terms is to shuffle the order of jobs running under the same allocation.

===How do we manage job priorities within our research group?===

'''Answer:'''

Obviously, managing shared resources within a large group - whether it
is conference funding or CPU time - takes some doing.

It's important to note that the fairshare periods are intentionally kept
quite short - just two weeks long. So, for example, let us say that in your resource
allocation you have about 10% of the machine. Then for someone to use
up the whole two week amount of time in 2 days, they'd have to use 70%
of the machine in those two days - which is unlikely to happen by
accident. If that does happen,
those using the same allocation as the person who used 70% of the
machine over the two days will suffer by having much lower priority for
their jobs, but only for the next 12 days - and even then, if there are
idle cpus they'll still be able to compute.

There will be online tools for seeing how the allocation is being used,
and those people who are in charge in your group will be able to use
that information to manage the users, telling them to dial it down or
up. We know that managing a large research group is hard, and we want
to make sure we provide you the information you need to do your job
effectively.

One way for users within a group to manage their priorities within the group
is with [[Moab#Adjusting_Job_Priority | user-adjusted priorities]]; this is
described in more detail on the [[Moab | Scheduling System]] page.

=== How do I charge jobs to my RAC allocation? ===

'''Answer:'''

Please see the [[Moab#Accounting|accounting section of Moab page]].

=== How does one check the amount of used CPU-hours in a project, and how does one get statistics for each user in the project? ===

'''Answer:'''

This information is available on the scinet portal,https://portal.scinet.utoronto.ca, See also [[SciNet Usage Reports]].

=== How does the Infiniband Upgrade affect my 2012 RAC allocation ?===

The RAC allocations for the current (2012) year that were based on ethernet and infiniband will carry over, however the allocation will be on the full GPC, not just the subsection. So if you were allocated 500 hours on Infiniband your fairshare allocation will still be 500 hours, just 500 out or 30,000, instead of 500 out of 7,000. If you received two allocations, one on gigE and one on IB, they will simply be combined. This should benefit all users as the desegregation of the GPC provides a greater pool of nodes increasing the probability of your job to run.

==Monitoring jobs in the queue==

===Why hasn't my job started?===

'''Answer:'''

Use the moab command

<pre>
checkjob -v jobid
</pre>

and the last couple of lines should explain why a job hasn't started.

Please see [[Moab| Job Scheduling System (Moab) ]] for more detailed information

===How do I figure out when my job will run?===

'''Answer:'''

Please see [[Moab#Available_Resources| Job Scheduling System (Moab) ]]



===I submit my GPC job, and I get an email saying it was rejected===
'''Answer:'''

This happens because the job you've submitted breaks one of the rules of the queues and is rejected. An email
is sent with the JOBID, JOBNAME, and the reason it was rejected. The following is an example where a job
requests more than 48 hours and was rejected.

<pre>
PBS Job Id: 3462493.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'wclimit too high for class 'batch_ib' (345600 > 172800)'
</pre>

Jobs on the TCS or GPC may only run for 48 hours at a time; this restriction greatly increases responsiveness of the queue and queue throughput for all our users. If your computation requires longer than that, as many do, you will have to [[ Checkpoints | checkpoint ]] your job and restart it after each 48-hour queue window. You can manually re-submit jobs, or if you can have your job cleanly exit before the 48 hour window, there are ways to [[ FAQ#How_can_I_automatically_resubmit_a_job.3F | automatically resubmit jobs ]].

Other rejections return a more cryptic error saying "job violates class configuration" such as follows:
<pre>
PBS Job Id: 3462409.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'user required by class 'batch''
</pre>

The most common problems that result in this error are:

* '''Incorrect number of processors per node''': Jobs on the GPC are scheduled per-node not per-core and since each node has 8 processor cores (ppn=8) the smallest job allowed is one node with 8 cores (nodes=1:ppn=8). For serial jobs users must bundle or batch them together in groups of 8. See [[ FAQ#How_do_I_run_serial_jobs_on_GPC.3F | How do I run serial jobs on GPC? ]]
* '''No number of nodes specified''': Jobs submitted to the main queue must request a specific number of nodes, either in the submission script (with a line like <tt>#PBS -l nodes=2:ppn=8</tt>) or on the command line (eg, <tt>qsub -l nodes=2:ppn=8,walltime=5:00:00 script.pbs</tt>). Note that for the debug queue, you can get away without specifying a number of nodes and a default of one will be assigned; for both technical and policy reasons, we do not enforce such a default for the main ("batch") queue.
* '''There is a 15 minute walltime minimum''' on all queues except debug and if you set your walltime less than this, it will be rejected.

===Running checkjob on my job gives me messages about JobFail and rejected===

Running checkjob on my job gives me messages that suggest my job has failed, as below: what did I do wrong?

<pre>
AName: test
State: Idle
Creds: user:xxxxxx group:xxxxxxxx account:xxxxxxxx class:batch_ib qos:ibqos
WallTime: 00:00:00 of 8:00:00
BecameEligible: Wed Jul 23 10:39:27
SubmitTime: Wed Jul 23 10:38:22
(Time Queued Total: 00:01:47 Eligible: 00:01:05)

Total Requested Tasks: 8

Req[0] TaskCount: 8 Partition: ALL
Opsys: centos6computeA Arch: --- Features: ---

Notification Events: JobFail

IWD: /scratch/x/xxxxxxxx/xxxxxxx/xxxxxxx
Partition List: torque,DDR
Flags: RESTARTABLE
Attr: checkpoint
StartPriority: 76
rejected for Opsys - (null)
rejected for State - (null)
rejected for Reserved - (null)
NOTE: job req cannot run in partition torque (available procs do not meet requirements : 0 of 8 procs found)
idle procs: 793 feasible procs: 0

Node Rejection Summary: [Opsys: 117][State: 2895][Reserved: 19]

NOTE: job violates constraints for partition SANDY (partition SANDY not in job partition mask)

NOTE: job violates constraints for partition GRAVITY (partition GRAVITY not in job partition mask)

rejected for State - (null)
NOTE:
</pre>

'''Answer:'''

The output from check job is a little cryptic in places, and if you are wondering why your job hasn't started yet, you might think that "rejection" and "JobFail" suggest that there's something wrong. But the above message is actually normal; you can use the <tt>showstart</tt> command on your job to get a (preliminary, subject to change) estimate as to when the job will start, and you'll find that it is in fact scheduled to start up in the near future.

In the above message:

* `Notification Events: JobFail` just means that, if notifications are enabled, you'll get a message if the job fails;
* `job req cannot run in partition torque` just means that the job cannot run just yet (that's why it's queued);
* `job req cannot run in dynamic partition DDR now (insufficient procs available: 0 < 8)` says why: there aren't processors available; and
* `job violates constraints for partition SANDY/GRAVITY` just means that the job isn't eligable to run in those paritcular (small) sections of the cluster.

that is, the above output is the normal and expected (if somewhat cryptic) explanation as to why the job is waiting - nothing to worry about.

===How can I monitor my running jobs on TCS?===

How can I monitor the load of TCS jobs?

'''Answer:'''

You can get more information with the command
/xcat/tools/tcs-scripts/LL/jobState.sh
which I alias as:
alias llq1='/xcat/tools/tcs-scripts/LL/jobState.sh'
If you run "llq1 -n" you will see a listing of jobs together with a lot of information, including the load.

==Errors in running jobs==



=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in
/var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

=== My GPC job died, telling me `Copy Stageout Files Failed' ===

'''Answer:'''

When a job runs on GPC, the script's standard output and error are redirected to
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:
<pre>
An error has occurred processing your job, see below.
request to copy stageout files failed on node
</pre>
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process and those files just never got copied.

Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.



===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or
network configurations may exhibit this problem at higher or lower MPI
task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

<pre>
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288
</pre>

With Intel MPI, you should be able to do

<pre>
module load intelmpi/4.0.3.008
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode
</pre>

to the same end.

For more information see [[GPC MPI Versions]].

===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===

'''Answer:'''

To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.

==Data on SciNet disks==

===How do I find out my disk usage?===

'''Answer:'''

The standard unix/linux utilities for finding the amount of disk space used by a directory are very slow, and notoriously inefficient on the GPFS filesystems that we run on the SciNet systems. There are utilities that very quickly report your disk usage:

The <tt>'''diskUsage'''</tt> command, available with the 'extras' module on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time.
This information is only updated hourly!

More information about these filesystems is available at the [[Data_Management | Data_Management]].

===How do I transfer data to/from SciNet?===

'''Answer:'''

All incoming connections to SciNet go through relatively low-speed connections to the <tt>login.scinet</tt> gateways, so using scp to copy files the same way you ssh in is not an effective way to move lots of data. Better tools are described in our page on [[Data_Management#Data_Transfer | Data Transfer]].

===My group works with data files of size 1-2 GB. Is this too large to transfer by scp to login.scinet.utoronto.ca ?===

'''Answer:'''

Generally, occasion transfers of data less than 10GB is perfectly acceptible to so through the login nodes. See [[Data_Management#Data_Transfer | Data Transfer]].

===How can I check if I have files in /scratch that are scheduled for automatic deletion?===

'''Answer:'''

Please see [[Storage_Quickstart#Scratch_Disk_Purging_Policy | Storage At SciNet]]

===How to allow my supervisor to manage files for me using ACL-based commands?===

'''Answer:'''

Please see [[Data_Management#File.2FOwnership_Management_.28ACL.29 | File/Ownership Management]]

===Can we buy extra storage space on SciNet?===

'''Answer:'''
Yes, please see [[Data_Management#Buying_storage_space_on_GPFS_or_HPSS | Buying storage space on GPFS or HPSS ]] for more details.

===Can I transfer files between BGQ and HPSS?===

'''Answer:'''
Yes, please see [https://support.scinet.utoronto.ca/wiki/index.php/BGQ#Bridge_to_HPSS Bridge to HPSS ] for more details.

==Keep 'em Coming!==

===Next question, please===

Send your question to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]; we'll answer it asap!

Introduction To Performance

2014-08-13T15:54:36Z

Ljdursi: /* Strong Scaling Tests */

==The Concepts of Parallel Performance==

Parallel computing used to be a very specialized domain; but now even making the best use of your laptop, which almost certainly has multiple independant computing cores, requires understanding the basic concepts of performance in a parallel environment.

Most fundamentally, parallel programming allows three possible ways of getting more and better science done:
;Running your computation many times
:If you have a program that works in serial, having many processors available to you allows you to run many copies of the same program at once, improving your [[#Throughput|throughput]]. This (can be) a sort of trivial use of parallel computing and doesn't require very specialized hardware, but it can be extremely useful for, for instance, running parameter studies or sensitivity studies. Best of all, this is essentially guaranteed to run efficiently if your serial code runs efficiently! Because this doesn't require fancy hardware, it is a waste of resources to use the [[TCS_Quickstart|Tightly Coupled System]] for these sorts of tasks and instead they must be run on the [[GPC_Quickstart|General Purpose Cluster]].
;Running your computation faster
:This is what most people think of as parallel computing. It can take a lot of work to make an existing code run efficiently on many processors, or to design a new code to make use of these resources, but when it works, one can achieve a substantial [[#Parallel_Speedup|speedup]] of individual jobs. This might mean the difference between a computation running in a feasible length of time for a research project or taking years to complete --- so while it may be a lot of work, it may be your only option. To determine whether your code runs well on many processors, you need to measure [[#Parallel_Speedup|speedup]] and [[#Efficiency|efficiency]]; to see how many processors one should use for a given problem you must run [[#Strong_Scaling_Tests|strong scaling tests]].
;Running your computation on larger problems
:One achieves speedup by using more processors on the same problem. But by running your job in parallel you may have access to more resources other than just processors --- for instance, more memory, or more disks. In this case, you may be able to run problems that simply wouldn't be possible on a single processor or a single computer; one can achieve significant '''''sizeup'''''. To find how large a problem one can efficiently run, one measures [[#Efficiency|efficiency]] and runs [[#Weak_Scaling_Tests|weak scaling tests]].

Of course, these aren't exclusive; one can take advantage of any combination of the above. It may be that your problem runs efficiently on 8 cores but no more; however, you may be able to get use of more processors by running many jobs to explore parameter space, and already on 8 cores you may be able to consider larger problems than you can with just one!

===Throughput===

Throughput is the most fundamental measure of performance, and the one that ultimately most matters to most computational scientists -- if you have N computations that you need to have done for your research project, how quickly can you get them done? Everything else we'll consider here is just a
way of increasing throughput T:
<math>
T = \frac{\mathrm{Number}\,\mathrm{of}\,\mathrm{computations}}{\mathrm{Unit}\,\mathrm{time}} .
</math>

If you have many indepdendent computations to perform (such as a parameter study or a sensitivity study) you can increase throughput almost arbitrarily by running them alongside each other at the same time, limited only by the number of processors available (or the wait time in the queue, or the disk space available, or some other external resource constraint). This approach obviously doesn't work if you only have one computation to perform, or if later
computations require the output from previous ones. In these cases, or when the individual jobs take infeasibly long, or cannot be performed on only one processor, one must resort to ''also'' using parallel programming techniques to parallelize the individual jobs.

===Compute Time===

Fundamental to everything else that follows is measuring the amount of time a computation takes on some problem size/amount of work <math>N</math> and some number of processors <math>P</math>. We'll denote this by <math>t(N,P)</math>. The easiest way to measure this time is with the <tt>time</tt> command that comes on most flavours of Unix in <tt>/bin/time</tt> or <tt>/usr/bin/time</tt>:

<pre>
/bin/time myprogram
...normal program output...
658.44user 0.85system 10:59.41elapsed
</pre>

The format of the times output at the end may vary from system to system, but the basic information returned will be the same. The ''real'' or ''elapsed'' time listed is the actual [[Wallclock time]] that elapsed during the run, ''user'' or ''cpu'' is the [[CPU time]] that was actually spent doing your computation, and the ''system'' time is the system time that was spent doing system-related things during the run, such as waiting for file input/output. Our goal will be to reduce the real wallclock time that the simulation takes as much as possible while still making efficient use of the resources available.

===Parallel Speedup===

The speedup of an individual job with some amount of work <math>N</math> as you go from some running it serially to running on <math>P</math> is simply:

<math>S(N,P) = \frac{t(N,P=1)}{t(N,P)} .</math>

That is, the time it takes to run the computation on P vs on one processor. The way this is usually done is to run the parallel code on <math>P</math> and
on <math>1</math> processor and take the ratio of the two times; but this is a form of cheating, as the parallel version of the code will generally have
overheads (even in the one-processor case) compared to the best available serial-only version of the code. The best thing to do in considering the efficiency of the parallelization to compare the parallel code to the best available serial code that does the same job.

If you are considering the speedup of a problem that doesn't fit onto one processor, of course, the concept of speedup can be generalized; one needn't start at <math>P=1</math>.

It should go without saying that, while developing your parallel code and during performance tuning, you get the same results with multiple processors as with some `known good' serial test case; it is even easier to introduce bugs in parallel code than it is in serial code!

===Efficiency===

Once you have a parallel code and some timing results one can look at how efficiently you are making use of the resources as you use more and more processors.
The parallel efficiency of a computation of some fixed work size running on <math>P</math> processors as compared to the <math>P=1</math> case is

<math>E = \frac{S(N,P)}{P}</math>

That is, if you get a speedup of <math>8 \times</math> in going from one to eight processors, you are at 1.00 or 100% efficiency; anything less and you are at lower efficiency. It isn't uncommon to achieve greater than 100% parallel efficiencies for small numbers of processors for some types of problems; as you go to more processors, you also have more processor cache, and thus more of the problems data can fit into fast cache. This is called ''super-linear speedup'' and sadly seldom extends out to very many processors.

===Strong Scaling Tests===

[[Image:scaling-example.png|thumb|right|320px|An example of a strong scaling test]]

The figure to the right and data below shows an example of a result of a small strong scaling test --- running a fixed-size problem on a varying number of processors to see how the timing of the computation scales with the number of processors. The code was an OpenMP code run on a node of the GPC. The quantitative results follow below; the times were measured and then speedups and efficiencies were calculated as above.

{|
! P
! t(N,P)
! S(N,P)
! E(N,P)
|-
| 1 || 3:50 || - || -
|-
| 2 || 2:02 || 1.87x || 94 %
|-
| 4 || 1:05 || 3.52x || 88 %
|-
| 6 || 47.8 || 4.81x || 80 %
|-
| 8 || 43.6 || 5.28x || 66%
|}

The plot shows the compute time <math>t(N,P)</math> as a function of P; if the code maintained 100% parallel efficiency, we would expect the scaling to be
as 1/P, so we plot it on a log-log scale. Also shown is the ideal scaling case -- what the times would be if, using the <math>P=1</math> timing as a normalization, we did get 100% efficiency. We can see that past 4 cores the measured case starts to significantly deviate from the ideal, and it looks like things would only get worse past 8 cores.

It's important to note here that scaling tests should be done on realistic problem sizes and for realistic lengths of time. Generally, for either serial or parallel programs there will be some overhead both at initialization time and during the course of the computation; if the problem size is too small, the overhead during the course of the run might be a significant fraction of the real work, and the program will behave needlessly poorly. Similarly, if the number of timesteps or iterations is too small, the initizalization overhead will similarly play a spuriously large role in the performance.

The above behaviour is typical for a small computation; it won't scale to too many cores, and the efficiency becomes monotonically worse as one increases the number of cores in use. The rate at which this happens will depend on the problem size and the type of computation. How is one to tell where to stop;
how good an efficiency is good enough? Certainly there are rules of thumb --- one shudders to see efficiencies below 50% --- but one can arrive at more meaningful and quantitative results by considering throughput. Let's imagine we had 64 cores at our disposal, and we wanted to run 96 jobs as quickly as possible. Our total time to completion of the 96 jobs would vary with the number of cores we ran per job as follows:

{|
! P
! Time for one job
! Time for all 96 jobs
|-
| 1 || 3:50 || 7:40 (2 batches, 64 jobs then 32)
|-
| 2 || 2:02 || 7:08 (3 batches, 32,32,32)
|-
| 4 || 1:05 || 6:30 (6 batches, 6x16)
|-
| 6 || 47.8 || 7:58 (10 batches, 9x10, 6)
|-
| 8 || 43.6 || 8:43 (12 batches)
|}

If we use more than 4 processes per job in this case, it will actually take us longer to do all our runs! For jobs that scale better with the number of processes (this could be a different program, or the same program with different problem size), we will find this turnover point to be at higher <math>P</math>; for jobs that scale worse, lower <math>P</math>.

===Weak Scaling Tests===

[[Image:weak-scaling-example.png|thumb|right|320px|An example of a weak scaling test]]

The strong scaling test described above considers the performance of a parallel code with a fixed work size as the number of processors varies; this tells us how the parallel overhead behaves as you go to more and more processors. A weak scaling test fixes the amount of work '''per processor''' and compares the execution time over number of processors. Since each processor has the same amount to do, in the ideal case the execution time should remain constant. While the strong scaling test tells you how the parallel overhead scales with <math>P</math>, the weak scaling test tells you something weaker -- whether the parallel overhead varies faster or slower than the amount of work.

Nonetheless, the weak scaling test can be the relevant one for determining how large a problem size one can efficiently compute with a given parallel code and system. An example of results for a weak scaling test on the GPC and TCS up to 256 processors (8 nodes of the TCS, 32 of the GPC) is shown to the right. In this case we are maintaining extremely good efficiency up to at least 128 processors with constant work per process on both architectures. It is possible to see different behaviour when first filling up a node (eg, for less than 8 processes for the GPC, or 64 for TCS) than when one starts crossing nodes; one should understand this but it doesn't necessarily indicate problems.

==Performance Tuning==

'''You cannot improve what you cannot measure.''' Performance tuning is an iterative process between running an '''instrumented''' version of your code, getting data on performance throughout the code, and attempting to make chances to the code that will make it run more efficiently.

There are three main ways of instrumenting a code to find its performance. The first is '''manually adding timers''' around important parts of the code to find out how much time is spent in each part. This is worth thinking about doing when putting together a new code, as it means that you'll have a very robust way of finding out how well the different parts of the code perform on different platforms and with different compiler options, etc.. The results are, however, necessarily very coarse-grained; they are very useful for comparing performance under different situations, but give very little information about whether or not there are performance problems or what they might be.

The second technique is '''sampling''', sometimes called `program counter sampling' or `statistical sampling'. In this case, the program is run in an environment where it is interrupted briefly at some set frequency (typically something like 100 times per second) and the location of the program counter is jotted down before the program is resumed. At the end of the program, these locations are translated into locations in the source code, and one has a statistical profile of where the program has spent its time.

Statistical sampling has several advantages. It has a very low overhead --- the sampling procedure for instance takes much less time than a function call to a timer routine --- so that the program runs much as it would without the measurement process. If the samples are taken often enough, the result is a very accurate picture of where your program is spending its time, allowing you to very quickly identify `hotspots' in the code and focus your attention on the most costly areas of the program. This combination of relevant information and low-overhead makes statistical sampling the first resort for serious performance measurement.

Sampling, however, has drawbacks. While it lets you know where the program is spending its time, it doesn't tell you why, or how it got there in the first place. For instance, in a parallel program you may be spending too much time in barriers of one sort or another (perhaps at <tt>MPI_WAITALL</tt> calls in MPI, or implicit barriers at the end of <tt>parallel</tt> sections in OpenMP) but unless you know where in the code that routine was called from, you can't address the problem. In this case you need some sort of '''trace''' through the program which keeps track of which routine called what. This is generally a much heavier-weight process, which can substantially increase the runtime of the code, running the risk of 'the Heisenburg effect' - measurement changing the system under observation. On the other hand, sometimes you just need that level of information, so tracing packages or libraries must be used.

A related method is the use of '''hardware counters''' --- counters within the CPU itself which keep track of performance-related information, such as the number of cache misses or branch mis-predictions within your code. Using this information, either regularly throughout the code or once for the entire code run can give very specific information about performance problems. Right now these counters are available on the TCS system but not on the GPC system, as the mainstream Linux kernel does not provide access to these counters.

===Simple Timer Wrappers: C, FORTRAN===

Below are some simple examples of timers in C and FORTAN which can be called before and after blocks of code to give wallclock times (in seconds) to give coarse-grained timings for sections of your code. Other approaches are possible.

Simple timers in C:
<source lang=c>
void tick(struct timeval *t) {
gettimeofday(t, NULL);
}

/* returns time in seconds from now to time described by t */
double tock(struct timeval *t) {
struct timeval now;
gettimeofday(&now, NULL);
return (double)(now.tv_sec - t->tv_sec) + ((double)(now.tv_usec - t->tv_usec)/1000000.);
}
</source>

and how to use them:
<source lang=c>
#include <sys/time.h>
struct timeval init, calc, io;
double inittime, calctime, iotime;

/*... */

tick(&init);
/* do initialization */
inittime = tock(&init);

tick(&calc);
/* do big computation */
calctime = tock(&calc);

tick(&io);
/* do IO */
iotime = tock(&io);

printf("Timing summary:\n\tInit: %8.5f sec\n\tCalc: %8.5f sec\n\tI/O : %8.5f sec\n",
inittime, calctime, iotime);
</source>

Simple timers in FORTRAN:

<source lang=fortran>
subroutine tick(t)
integer, intent(OUT) :: t

call system_clock(t)
end subroutine tick

! returns time in seconds from now to time described by t
real function tock(t)
integer, intent(in) :: t
integer :: now, clock_rate

call system_clock(now,clock_rate)

tock = real(now - t)/real(clock_rate)
end function tock
</source>

And using them:
<source lang=fortran>
call tick(calc)
! do big calculation
calctime = tock(calc)

print *,'Timing summary'
print *,'Calc: ', calctime
</source>

==Command-line Performance Tools==

Many of the tools below can be used to examine both serial and parallel performance problems with a code. We'd like to encourage you to tune serial performance first. Worrying about parallel performance before the code performs well with a single task doesn't make much sense! Profiling your code when running with one task allows you to spot serial `hot spots' for optimization, as well as giving you more detailed understanding of where your program spends its time. Further, any performance you make in the serial code will automatically speed up your parallel code.

We've already talked about coarse-grained measurements such as timers within the code and using tools such as <tt>/bin/time</tt>. These are very useful for comparing overall performance between different platforms/parameters, but we won't need to discuss them further here.

===gprof (profiling: everywhere)===

A statistical sampling workhorse is <tt>gprof</tt>, the GNU version of an old common Unix utility called prof. To use this, the code must be re-compiled with both source-code symbols intact (<tt>-g</tt>) and with profiling information available (for most compilers, this is <tt>-pg</tt>; for the IBM compilers (xlf, xlc, xlC) it is <tt>-p</tt>). It is worth knowing because of its ubiquity, and because it contains much of the functionality of newer tools so that the same concepts occur in other concepts.

So let's consider the following trivial program <tt>pi.c</tt>:
<source lang="c">
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

double calc_pi(long n) {
long in = 0;
long out = 0;
long i;
double x,y;

for (i=0; i<n; i++) {
x = drand48();
y = drand48();
if (x*x+y*y < 1) {
in++;
} else {
out++;
}
}

return 4.*(double)in/(double)(in+out);
}

int main(int argc, char **argv) {
long n, defaultn=100000;
double pi;
time_t t;

/* seed random number generator */
srand48(time(&t));

/* get number of tries */
if (argc < 2 || (n=atoi(argv[1]))<1) {
n = defaultn;
printf("Using default n = %ld\n", n);
}

pi = calc_pi(n);
printf("Pi = %lf\n", pi);

return 0;
}
</source>

We can compile this with profiling on and run it:
<pre>
$ gcc -g -pg -o pi pi.c
$ ./pi 100000000
Pi = 3.141804

</pre>

(Note that this isn't a very good way of calculating pi!). On exit, this program creates a file called <tt>gmon.out</tt>; this contains the profiling information about the run of the code. We can take a look at this by using <tt>gprof</tt>:
<pre>
$ gprof pi gmon.out
Flat profile:

Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls ms/call ms/call name
100.88 1.00 1.00 1 998.76 998.76 calc_pi

index % time self children called name
1.00 0.00 1/1 main [2]
[1] 100.0 1.00 0.00 1 calc_pi [1]
-----------------------------------------------
<spontaneous>
[2] 100.0 0.00 1.00 main [2]
1.00 0.00 1/1 calc_pi [1]
-----------------------------------------------
</pre>

The first part tells us that essentially all of the time spent running was in the <tt>calc_pi()</tt> routine (of course), and the second part attepts to be a call graph, showing that <tt>main</tt> called <tt>calc_pi()</tt> once. An important concept in the timing is the `self' and `children' times for each routine, sometimes called the inclusive and exclusive time. Because most routines call many other routines, its often useful to distinguish between the total amount of time spent between starting and ending the routine (the `inclusive' time) and that same time excluding the time spent in child routines (`exclusive' time).

The above results are fairly trivial and not very useful for this simple program, but in more complicated routines it can be very valuable to narrow down hotspots to particular regions of code.

[[Image:Xprofiler.png|thumb|300px|The AIX tool <tt>Xprof</tt> gives a visual representation of the <tt>gprof</tt> output.]]

In fact, gprof also allows you to view the time spent in the code by lines of code. As you chop the program up finer, the statistical sampling gets less accurate; thus to look at the results by line of code you must be sure that your sample run was long enough to get meaningful data. But the results can be extremely useful:
<pre>
$ gprof --line pi gmon.out
Flat profile:

Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls Ts/call Ts/call name
70.31 0.70 0.70 calc_pi (pi.c:14 @ 40078b)
14.27 0.84 0.14 calc_pi (pi.c:17 @ 4007bc)
5.10 0.89 0.05 calc_pi (pi.c:11 @ 4007c1)
4.08 0.93 0.04 calc_pi (pi.c:15 @ 4007b5)
3.06 0.96 0.03 calc_pi (pi.c:13 @ 400781)
2.55 0.98 0.03 calc_pi (pi.c:12 @ 400777)
1.53 1.00 0.02 calc_pi (pi.c:11 @ 40076d)
0.00 1.00 0.00 1 0.00 0.00 calc_pi (pi.c:5 @ 40074c)
</pre>

where now we can see that the single line containing the radius calculation (<tt>if (x*x+y*y < 1)</tt>) is 70% of the work for the entire program. This tells you where you should spend your time to optimize the code. Other tools exist for this sort of line-by-line analysis; <tt>gcov</tt> in the gcc compiler suite counts the number of times a given source line is executed - the idea was for coverage analysis for test suites, but it certainly can be used for profiling as well; however, usually the amount of time spent at a line is more important than the number of executions.

For parallel programs, <tt>gprof</tt> will generally output a seperate <tt>gmon.out</tt> file for each process; for threaded applications, output for all threads will be summed into the same <tt>gmon.out</tt>. It may be useful to sum up all the results and view them with gprof or to look at them individually.

There are other tools for looking at the same data. For instance, on the TCS system, the command <tt>Xprof</tt>
(run the same way as <tt>gprof</tt>; <tt>Xprof program_name gmon.out</tt>) lets you look at the call tree as a graphical tree. Each routine is shown by a block with a size proportional to the time spent in each routine; the width is the inclusive time, and the height is the exclusive time.

===hpmcount (performance counters: TCS)===

On the TCS, <tt>hpmcount</tt> allows the querying of the performance counter values over the course of a run. Since here we are simply asking the CPU to report values it obtains during the run of a program, the code does not need to be instrumented; simply typing
<pre>
hpmcount hpmcount_args program_name program_args
</pre>

will run the program and output the results from the hardware performance counters at the end. So for instance, with our trivial pi program above,

<pre>
tcs-f11n05-$ hpmcount ./pi
Using default n = 100000
Pi = 3.144240
Execution time (wall clock time): 0.020325 seconds

######## Resource Usage Statistics ########

Total amount of time in user mode : 0.012754 seconds
Total amount of time in system mode : 0.001486 seconds
Maximum resident set size : 440 Kbytes
Average shared memory use in text segment : 0 Kbytes*sec
Average unshared memory use in data segment : 0 Kbytes*sec
Number of page faults without I/O activity : 53
Number of page faults with I/O activity : 1
Number of times process was swapped out : 0
Number of times file system performed INPUT : 0
Number of times file system performed OUTPUT : 0
Number of IPC messages sent : 0
Number of IPC messages received : 0
Number of signals delivered : 0
Number of voluntary context switches : 6
Number of involuntary context switches : 0

####### End of Resource Statistics ########

Set: 1
Counting duration: 0.014947083 seconds
PM_FPU_1FLOP (FPU executed one flop instruction ) : 400093
PM_FPU_FMA (FPU executed multiply-add instruction) : 500030
PM_FPU_FSQRT_FDIV (FPU executed FSQRT or FDIV instruction) : 1
PM_CYC (Processor cycles) : 58485795
PM_RUN_INST_CMPL (Run instructions completed) : 24238152
PM_RUN_CYC (Run cycles) : 70307511

Utilization rate : 61.172 %
Flop : 1.400 Mflop
Flop rate (flops / WCT) : 68.888 Mflop/s
Flops / user time : 112.614 Mflop/s
FMA percentage : 111.103 %

</pre>
There are a variety of sets of performance counters that can be reported; the default set isn't especially helpful for HPC-type computations; sets of performance counters can be specified on the commandline in the format <tt>-d -s item,item,item</tt>. Sets 5 and 12 are very useful for showing memory performance (showing L1 and L2 cache misses) and set 6 is especially useful for shared memory profiling, giving statistics about how often off-processor memory had to be accessed.

Showing the counters for the entire program will often tell you if there's a problem or not, but won't tell you where it is. For more detailed information, one can [http://www.ncsa.uiuc.edu/UserInfo/Resources/Software/Tools/HPMToolkit/HPM_2_5_2.AIX.html use the hpm library] to manually instrument different regions of your code, and get similar outputs to above for several different, smaller, regions of code.

On the linux side, <tt>oprofile</tt> allows the reporting of similar information, but to use it one must have root access to the linux machine.

===cachegrind (Memory use analysis: GPC)===

[[Image:Kcachegrind.png|thumb|kcachegrind, part of the KDE development package, can give graphical overviews of the output from cachegrind]]

<tt>[http://valgrind.org/ valgrind]</tt> is a memory tool that is usually thought of in terms of finding memory-access bugs in large programs. Rather than instrumenting a code or measuring counters, valgrind takes a fairly extreme approach -- it emulates your program running on a computer, essentially running a simulation of your program running on the same kind of computer valgrind is running on. This has enormous overhead (runtimes can be up to 20x as long as normal) but the result is exquisitely detailed information about what your program is doing.

Memory access is often a bottleneck for HPC codes, and cachegrind is a tool for valgrind which simulates the use of cache in your program, giving you line-by-line information on which parts of the code have cache performance issues. Your code does not need to be recompiled, although compiling with <tt>-g</tt> is necessary for the output to be useful. Cachegrind is run as shown:
<pre>
valgrind --tool=cachegrind myprogram myprogram_arguments
</pre>

Overall results for the whole program are given at the end of the programs normal output, and more detailed information is saved in a file that begins with <tt>cachegrind.out</tt>. These output files are XML files - readable in principle by humans, but it is much easier to see what is going on with visual tools like kcachegrind (shown to the right) or, eventually, valkyrie (which also can be used for <tt>memcheck</tt> output.)

===IPM (MPI Tracing and hardware counters: GPC, TCS)===

[[Image:IPM.png|thumb|IPM generates a series of webpages and graphs summarizing performance of your code which can then be viewed in a web browser]]

IPM is the [Integrated Performance Monitors http://ipm-hpc.sourceforge.net/] which monitor a variety of MPI and hardware performance information. There are a number of modules for IPM (Integrated Performance Monitors)
on the TCS and GPC depending on which machine and compilers/MPI you use.

These can be linked in to your MPI executable at link- or run-time, and generate
detailed output at the end of your run which can be parsed and produce
a nice set of HTML + graphics.

Running IPM varies slightly on the GPC and the TCS. In the GPC, because the default for the MPI libraries is to be
compiled in as dynamic, shared libraries, it is easiest just to compile your program as normal and link in these
libraries only when you are about to run your program. In your submission script, run your program as so:

<pre>
module load ipm/[appropriate-version]
export LD_PRELOAD=${SCINET_IPM_LIB}/libipm.so
mpirun [...]
export LD_PRELOAD=
</pre>

On the other hand, on TCS, it is easist to link in the IPM libraries at link time, with
<pre>
...-L${SCINET_IPM_LIB} -lipm
</pre>

Either way, once the once the program is finished, in the directory it is run in
there will be a large XML file with an ugly name like

<pre>[username].[longnumber]</pre>

to turn this into useful data, do:
<pre>
$ module load ipm/[appropriate-version]
$ ipm_parse -html [username].[longnumber]
</pre>

That will produce a directory named something like
<pre>[executable].[username].[number].[node]
</pre>
which you can copy back to your
computer and view the index.html file within for lots of detailed info about your run.

Note that on the GPC, there are IPM modules for "posix" and "mpiio". Which of these you use only matters if you want to try to do I/O profiling
as well as MPI profiling. If you do want to do IO tracing, you will have to be sure to choose the right module variant (mpiio if you do parallel I/O, either manually with MPI-I/O or through a parallel HDF5 or NetCDF library, and posix otherwise) and statically link the library into your executable at compile time:
<pre>
...-L${SCINET_IPM_LIB} -lipm
</pre>

==Graphical Performance Tools==

While graphical performance tools typically measure similar things to their command-line relations, a graphical display opens up the possibility of aggregating much more information and having increased flexibility in displaying it in a variety of ways; this can be very helpful, especially in the initial stages of finding performance problems.

===OpenSpeedShop (profiling, MPI tracing: GPC)===

[[Image:Speedshop2.png|thumb|OpenSpeedShop, like gprof will tell you where the hotspots are in the code by function]]
[[Image:Speedshop1.png|thumb|...or by line of code]]

[http://www.openspeedshop.org OpenSpeedShop] is a tool that is installed on GPC; it is currently compiled for support only with gcc and openmpi. To use it,
<pre>
module load gcc python intel openmpi openspeedshop
</pre>
It provides the functionality of gprof, with the addition of hardware counter measurements (not currently supported on GPC machines) and options to do both lightweight and more detailed, more heavy-weight profiling. OpenSpeedShop also contains enhanced support for dealing with parallel runs, and for tracing of MPI or I/O calls to find performance problems to those areas. The parallel support is considerably more than what <tt>gprof</tt> offers; bundling the data from thousands of tasks into one set of results is a significant algorithmic challenge in itself.

Another important addition, shared by many of the other graphical tools, is the idea of bundling results into different `experiments' --- bundles of an executable, measurement type, and resulting data --- which makes the iterative process of performance tuning much easier. OpenSpeedShop, as with some other tools, has the ability to directly compare the results of different experiments, so one can more easily see if a particular change made things better or worse, and if so where.

OpenSpeedshop does not require re-compilation of the executable (although as with all these tools, for the correlation with the source code to be useful, the code should be compiled with debugging symbols, the option for which is almost universally <tt>-g</tt> to the compiler and linker. The code is then either instrumented, or run in an instrumented environment. Shown to the right are two of the views available for exmaining the timing results of one OpenMP code.

OpenSpeedShop can be launched from the commandline and then used entirely through the gui: there are a variety of `wizards' which guide you through choosing how to instrument and run your experiment:

<pre>
$ openss
</pre>

this should be done from a directory containing the source code and the executable. This is an excellent way to get started with this tool. Once one is more familiar with the tool, one can run a variety of experiments on the command line:

<pre>
openss –f program_name pcsamp
</pre>

where the above runs the <tt>pcsamp</tt> (program counter sampling, as in gprof) measurement on the executable <tt>program_name</tt>. Then one can launch the gui to view the results. There are options for instrumenting the executable in a variety of ways, and taking different measurements; the [http://www.openspeedshop.org OpenSpeedShop] web page contains links to documentation and tutorials.

===PeekPerf (profiling, TCS)===

[[Image:PeekPerf.png|thumb|An example of using PeekPerf]]

[http://domino.research.ibm.com/comm/research_projects.nsf/pages/hpct.index.html Peekperf] is IBM's single graphical `dashboard' providing access to many performance measurement tools for exmaining Hardware Counter data, threads, message passing, IO, and memory access, several of which are available seperately as command-line tools. Like OpenSpeedShop, it does not require re-compilation of the executable; an instrumented version of the code is generated at run time and this instrumented version is executed with whatever options you care to pass to it. It does not have the same support for comparing experiments as OpenSpeedShop does, however it allows running several different types of measurements at once, seeing how they correlate in a given run; this is something OpenSpeedShop doesn't have.

One starts peekperf at the commandline
<pre>
$ peekperf
</pre>

and tell peekperf which executableyou wish to run measurements on. You then highlight which sorts of measurements you wish to make (which sorts are available depend on the type of program - threaded, OpenMPed, etc), select `generate an instrumented executable', and then `run the instrumented executable', giving it the name of either the instrumented executable or a script to run the same; the program will then begin displaying the resulting data as soon as the program has completed.

Understanding the interface and resulting data takes some practice, and the documentation is quite sparse; however the flexibility in the range of measurements to take makes this an excellent source of performance information for programs running on the TCS system.

===Scalasca (profiling, tracing: TCS, GPC)===
[[Image:Scalasca.png|thumb|An example of using Scalasca]]

[http://www.scalasca.org Scalasca] is a sophisticated tool which takes the aggregation of data shown in the above graphical tools one step further and analyzes the results to pinpoint and display common performance problems; it scales extremely well and the graphical display makes it very easy for the user to find out where the performance issues are. To use scalasca, on either the TCS or the GPC load the scalasca module:
<pre>
$ module load scalasca
</pre>

Scalasca requires the code to be recompiled, and it has wrapper scripts to select and choose the right options to use. If your code for instance is normally compiled with
<pre>
ifort -c myprog.f
ifort -o myprog myprog.o -lm
</pre>

then one can instead use
<pre>
scalasca -instrument ifort -c myprog.f
scalasca -instrument ifort -o myprog myprog.o -lm
</pre>

Note that with the intel compilers, if you're using OpenMP, you'll have to add the -pomp flag, eg <tt>scalasca -instrument -pomp</tt>.

Scalasca then parses the rest of the command line and adds the necessary flags. (If you are curious, <tt>scalasca -instrument -v</tt> will show you what the resulting command line actually is.) There is also a shortcut, <tt>skin</tt>, which is equivalent to <tt>scalasca -instrument</tt>.

When the new executable is generated, it's run in a similar way; if you normally run your program as
<pre>
./myprog
</pre>

or
<pre>
mpirun -np 5 ./myprog
</pre>

You'd instead do
<pre>
scalasca -analyze ./myprog
scalasca -analyze mpirun -np 5 ./myprog
</pre>

The program will run as usual with only additional lines about output to files. (Again, there is a shortcut available; <tt>scan</tt> is equivalent to <tt>scalasca -analyze</tt>.) To then look at the results with a graphical user interface, one uses
<pre>
scalasca -examine [epik directory name]
</pre>
where the directory name is that created by the analyze program. This tries to pop up an xwindow; from an xterm on a linux or Mac machine, you will have to login via
<pre>
$ ssh -Y login.scinet.utoronto.ca
scinet01$ ssh -Y [whichever devel node you're using]
</pre>
and you will have to run this from one of the devel nodes, rather than one of the compute nodes.

A screenshot of the results is shown to the right for an OpenMP program, where wait times at implicit barriers at the end of parallel sections is selected as an important metric to show on the left; the middle panel shows the call tree indicating the context in which the delays occurred, and the panel on the right gives the breakdown for each thread.

<!-- ===VTune/Thread Profiler (GPC)=== don't have license up for this yet --!>

==Common Serial Performance Problems==

===Poor use of cache===
A classic problem for scientific codes is memory bandwidth; the capacity to do on-chip floating-point or integer operations has grown much faster than the ability to get numbers onto the chip in the first place. One way around that is to use various levels of memory cache; when one number is needed from memory, a whole line of data is brought in from (slow) external memory to fast on-chip cache. This makes the memory access modestly slower, but tends to greatly speed up performance since if you are going to do something to data in one part of memory you're typically going to be also doing to neighboring values.

If you take advantage of data locality --- accessing memory in some kind of order rather than jumping around in memory --- cache can greatly increase the performance of your code. On the other hand, if you '''do''' jump around in memory a lot, cache will actually hurt your performance.

The classic way this comes up is in accessing multidimensional arrays. The example below is simplified; most cases aren't this extreme (or obvious!) but the idea is the same. Let's consider the following FORTRAN code, which simply iterates a few time through a modestly sized multidimensional array:
<source lang="f">
program memaccess

integer, parameter :: li=32,lj=32,lk=32,ll=32,lm=32
real, dimension(li,lj,lk,ll,lm) :: a
integer :: i,j,k,l,m
integer :: iter

a = 0.

do iter=1,10
do m=1,lm
do l=1,ll
do k=1,lk
do j=1,lj
do i=1,li
a(i,j,k,l,m) = a(i,j,k,l,m)+ i+j+k+l+m
enddo
enddo
enddo
enddo
enddo
enddo

end program
</source>
The above program, which we'll suggestively call <tt>memaccess-good.f</tt>, accesses array elements in the order that FORTRAN places in them in the computer's memory; FORTRAN lays out this array in memory as <tt>[a(1,1,1,1,1), a(2,1,1,1,1),... a(31,1,1,1,1), a(1,2,1,1,1)...]</tt> and so on. So by ordering our loops that way we are marching through memory in order, making maximum use of cache. The resulting code can be timed:
<pre>
$ gfortran -O3 -o memaccess-good memaccess-good.f
$ time ./memaccess-good

real 0m2.478s
user 0m2.337s
sys 0m0.094s
</pre>

If we reverse the order of the loops, so they go <tt> do i=1,li.. do j=1,lj,..... do m=1,lm</tt>, however, we get
<pre>
$ gfortran -O3 -o memaccess-bad memaccess-bad.f
$ time ./memaccess-bad

real 0m19.622s
user 0m19.101s
sys 0m0.098s
</pre>
A factor of 8 times worse! Thus tools such as cachegrind can be extremely important for finding significant performance problems in memory-heavy codes.

C-based languages arrange their arrays the opposite ways in memory, so that the equivalent array in C would go as <tt>[a[0][0][0][0][0], a[0][0][0][0][1], ... a[0][0][0][0][31], a[0][0][0][1][0], ... ]</tt>; thus `bad' array access in FORTRAN looks like `good' array access in C, and vice versa.

==Common OpenMP Performance Problems==

==Common MPI Performance Problems==
===Overuse of MPI_BARRIER===
===Many Small Messages===
Typically, a the time it takes for a message of size ''n'' to get from one node to another can be expressed in terms of a [[latency]] ''l'' and a [[bandwidth]] ''b'',
<math>t_c = l + \frac{n}{b} .</math>
For small messages, the latency can dominate the cost of sending (and processing!) the message. By
bundling many small messages into one, you can amortize that cost over many messages, reducing
the time spent communicating.
===Not overlapping computation and communications===

User Serial

2014-07-30T20:10:35Z

Ljdursi: /* Use a whole node... */

===General considerations===

====Use whole nodes...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconveniencing you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20140622
</source>
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version.

The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>. SciNet's gnu-parallel modules automatically transfer <tt>OMP_NUM_THREADS</tt>, and typical environment variables set by most modules.

Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T20:10:10Z

Ljdursi: /* ...but not more. */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconveniencing you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20140622
</source>
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version.

The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>. SciNet's gnu-parallel modules automatically transfer <tt>OMP_NUM_THREADS</tt>, and typical environment variables set by most modules.

Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:25:50Z

Ljdursi: /* Version for more than 8 cores at once (still serial) */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconvenicning you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20140622
</source>
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version.

The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>. SciNet's gnu-parallel modules automatically transfer <tt>OMP_NUM_THREADS</tt>, and typical environment variables set by most modules.

Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:23:25Z

Ljdursi: /* Version for more than 8 cores at once (still serial) */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconvenicning you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20140622
</source>
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version.

The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>.
Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:21:48Z

Ljdursi: /* Serial jobs of varying duration */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconvenicning you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20140622
</source>
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version.

The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>.
Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:20:47Z

Ljdursi: /* GNU Parallel */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconvenicning you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20140622
</source>
Note that there are several versions of gnu-parallel installed on the GPC; we recommend using the newer version.

The citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

module load gnu-parallel/20140622

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>.
Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:19:12Z

Ljdursi: /* Serial jobs of similar duration */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconvenicning you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# Turn off implicit threading in Python, R
export OMP_NUM_THREADS=1

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20130422
</source>
Note that there are currently (May 2013) four versions of gnu-parallel installed on the GPC, with the older version, gnu-parallel/2010, as the default, although we'd recommend using the newer version.

Note that the citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>.
Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:18:37Z

Ljdursi: /* General considerations */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If they use more than this, they will crash the node, inconvenicning you and other researchers waiting for that node.

If that's not possible -- each individual job requires significantly in excess of ~1.75GB -- then it's possible to just run fewer jobs so that they do fit; but then, again there is an under-utilization problem. In that case, the jobs are likely candidates for parallelization, and you can contact us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores for certain types of tasks has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

While your program may not be explicitly parallel, it may use some of SciNet's threaded libraries for numerical computations, which can make use of multiple processors. In particular, SciNet's [[Python]] and [[R_Statistical_Package | R]] modules are compiled with aggressive optimization and using threaded numerical libraries which by default will make use of multiple cores for computations such as large matrix operations. This can greatly speed up individual runs, but by less (usually much less) than a factor of 8. If you do have many such computations to do, your [[Introduction_To_Performance#Throughput | throughput]] will be better - you will get more calculations done per unit time -if you turn off the threading and run multiple such computations at once. Threading is turned off with the shell script line <tt>export OMP_NUM_THREADS=1</tt>; that line will be included in the scripts below.

If your calculations do implicitly use threading, you may want to experiment to see what gives you the best performance - you may find that running 4 (or even 8) jobs with 2 threads each (<tt>OMP_NUM_THREADS=2</tt>), or 2 jobs with 4 threads, gives better performance than 8 jobs with 1 thread (and almost certainly better than 1 job with 8 threads). We'd encourage to you to perform exactly such a [[Introduction_To_Performance#Strong_Scaling_Tests | scaling test]]; for a small up-front investment in time you may significantly speed up all the computations you need to do.

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20130422
</source>
Note that there are currently (May 2013) four versions of gnu-parallel installed on the GPC, with the older version, gnu-parallel/2010, as the default, although we'd recommend using the newer version.

Note that the citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>.
Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

User Serial

2014-07-30T19:05:56Z

Ljdursi: /* General considerations */

===General considerations===

====Use a whole node...====

When you submit a job on a SciNet system, it is run on one (or more than one) entire node - meaning that your job is occupying at least 8 processors during the duration of its run. The SciNet systems are usually, with many researchers waiting in the queue for computational resources, so we require that you make full use of the nodes that your job is allocated, so other resarchers don't have to wait unnecessarily, and so that your jobs get as much work done for you while they run as possible.

Often, the best way to make full use of the node is to run one large parallel computation; but sometimes it is beneficial to run several serial codes at the same time. On this page, we discuss ways to run suites of serial computations at once, as efficiently as possible, using the full resources of the node.

====...but not more.====

When running multiple jobs on the same node, it is essential to have a good idea of how much memory the jobs will require. The GPC compute nodes have about 14GB in total available
to user jobs running on the 8 cores (a bit less, say 13GB, on the devel ndoes <tt>gpc01..04</tt>, and [[GPC_Quickstart#Memory_Configuration|somewhat more for some compute nodes]])
So the jobs also have to be bunched in ways that will fit into 14GB. If that's not possible --
each individual job requires significantly in excess of ~1.75GB -- then
its possible in principle to just run fewer jobs so that they do fit;
but then, again there is an under-utilization problem. In that case,
the jobs are likely candidates for parallelization, and you can contact
us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] and arrange a meeting with one of the
technical analysts to help you do just that.

If the memory requirements allow it, you could actually run more than 8 jobs at the same time, up to 16, exploiting the [[GPC_Quickstart#HyperThreading | HyperThreading]] feature of the Intel Nehalem cores. It may seem counterintuitive, but running 16 jobs on 8 cores has increased some users overall throughput by 10 to 30 percent.

====Is your job really serial?====

===Serial jobs of similar duration===

The most straightforward way to run multiple serial jobs is to bunch the jobs in groups of 8 or more that will take roughly the same amount of time, and create a job that looks a
bit like this
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialx8

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; ampersand off 8 jobs and wait
(cd jobdir1; ./dojob1) &
(cd jobdir2; ./dojob2) &
(cd jobdir3; ./dojob3) &
(cd jobdir4; ./dojob4) &
(cd jobdir5; ./dojob5) &
(cd jobdir6; ./dojob6) &
(cd jobdir7; ./dojob7) &
(cd jobdir8; ./dojob8) &
wait
</source>

There are four important things to take note of here. First, the <tt>'''wait'''</tt>
command at the end is crucial; without it the job will terminate
immediately, killing the 8 programs you just started.

Second is that it is important to group the programs by how long they
will take. If (say) <tt>dojob8</tt> takes 2 hours and the rest only take 1,
then for one hour 7 of the 8 cores on the GPC node are wasted; they are
sitting idle but are unavailable for other users, and the utilization of
this node over the whole run is only 56%. This is the sort of thing
we'll notice, and users who don't make efficient use of the machine will
have their ability to use scinet resources reduced. If you have many serial jobs of varying length,
use the submission script to balance the computational load, as explained [[ #Serial jobs of varying duration | below]].

Third, we reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs.

===GNU Parallel===

GNU parallel is a really nice tool written by Ole Tange to run multiple serial jobs in
parallel. It allows you to keep the processors on each 8core node busy, if you provide enough jobs to do.

GNU parallel is accessible on the GPC in the module
<tt>gnu-parallel</tt>:
<source lang="bash">
module load gnu-parallel/20130422
</source>
Note that there are currently (May 2013) four versions of gnu-parallel installed on the GPC, with the older version, gnu-parallel/2010, as the default, although we'd recommend using the newer version.

Note that the citation for GNU Parallel is: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

It is easiest to demonstrate the usage of GNU parallel by
examples. Suppose you have 16 jobs to do, that these jobs duration varies quite a bit, but that the average job duration is around 10 hours. You could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=24:00:00
#PBS -N gnu-parallel-example

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# EXECUTION COMMAND
parallel -j 8 <<EOF
cd jobdir1; ./dojob1; echo "job 1 finished"
cd jobdir2; ./dojob2; echo "job 2 finished"
cd jobdir3; ./dojob3; echo "job 3 finished"
cd jobdir4; ./dojob4; echo "job 4 finished"
cd jobdir5; ./dojob5; echo "job 5 finished"
cd jobdir6; ./dojob6; echo "job 6 finished"
cd jobdir7; ./dojob7; echo "job 7 finished"
cd jobdir8; ./dojob8; echo "job 8 finished"
cd jobdir9; ./dojob9; echo "job 9 finished"
cd jobdir10; ./dojob10; echo "job 10 finished"
cd jobdir11; ./dojob11; echo "job 11 finished"
cd jobdir12; ./dojob12; echo "job 12 finished"
cd jobdir13; ./dojob13; echo "job 13 finished"
cd jobdir14; ./dojob14; echo "job 14 finished"
cd jobdir15; ./dojob15; echo "job 15 finished"
cd jobdir16; ./dojob16; echo "job 16 finished"
EOF
</source>

The <tt>-j8</tt> parameter sets the number of jobs to run at the same time, but 16 jobs are lined up. Initially, 8 jobs are given to the 8 processors on the node. When one of the processors is done with its assigned job, it will get a next job instead of sitting idle until the other processors are done. While you would expect that on average this script should take 20 hours (each processor on average has to complete two jobs of 10hours), there's a good chance that one of the processors gets two jobs that take more than 10 hours, so the job script requests 24 hours. How much more time you should ask for in practice depends on the spread in run times of the separate jobs.

===Serial jobs of varying duration===

If you have a lot (50+) of relatively short serial runs to do, '''of which the walltime varies''', and if you know that eight jobs fit in memory without memory issues, then writing all the command explicitly in the jobscript can get tedious. If you follw a convention in that the jobs are all started by auxiliary scripts called jobs<something>, the following strategy in your submission script would maximize the cpu utilization.
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N serialdynamic

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# COMMANDS ARE ASSUMED TO BE SCRIPTS CALLED job*.sh
echo job*.sh | tr ' ' '\n' | parallel -j 8
</source>
Notes:
* As before, GNU Parallel keeps 8 jobs running at a time, and if one finishes, starts the next. This is an easy way to do ''load balancing''.
* You can in fact run more or less than 8 processes per node by modifying <tt>parallel</tt>'s <tt>-j8</tt> argument.
* Doing many serial jobs often entails doing many disk reads and writes, which can be detrimental to the performance. In that case, running from the ramdisk may be an option.
* When using a ramdisk, make sure you copy your results from the ramdisk back to the scratch after the runs, or when the job is killed because time has run out.
* More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].
* This script optimizes resource utility, but can only use 1 node (8 cores) at a time. The next section addresses how to use more nodes.

===Version for more than 8 cores at once (still serial)===

If you have hundreds of serial jobs that you want to run concurrently and the nodes are available, then the approach above, while useful, would require tens of scripts to be submitted separately. It is possible for you to request more than one node and to use the following routine to distribute your processes amongst the cores. In this case, it is important to use the newer version of GNU parallel installed on the GPC.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for multiple, dynamically-run
# serial jobs on SciNet GPC
#
#PBS -l nodes=25:ppn=8,walltime=1:00:00
#PBS -N serialdynamicMulti

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

module load gnu-parallel/20130422

# START PARALLEL JOBS USING NODE LIST IN $PBS_NODEFILE
seq 800 | parallel -j8 --sshloginfile $PBS_NODEFILE --workdir $PWD ./myrun {}
</source>
Explanation:
* <tt>seq 800</tt> outputs the numbers 1 through 800 on separate lines. This output is piped to (ie becomes the input of) the <tt>parallel</tt> command.
* The use of the "seq 800" is that each line that you give to <tt>parallel</tt> defines a new job. So here, there are 800 jobs.
* Each job runs a command, but because the numbers generated by seq are not commands, a real command is constructed, in this case, by the argument <tt>./myrun {}</tt>. Here <tt>myrun</tt> is supposed to be the name of the application to run. The two curly brackets <tt>{}</tt> get replaced by the line from the input, that is, by one of the numbers.
* So parallel will run the 800 commands: ./myrun 1 ./myrun 2 ... ./myrun 800
* The parameter <tt>--sshloginfile $PBS_NODEFILE</tt> tells <tt>parallel</tt> to look for the file named $PBS_NODEFILE which contains the host names of the nodes assigned to the current job (as stated above, it is automatically generated).
* The parameter <tt>-j8</tt> tells <tt>parallel</tt> to run 8 of these at a time on each of the hosts.
* The <tt>--workdir $PWD</tt> sets the working directory on the other nodes to the working directory on the first node. Without this, the run tries to start from the wrong place and will most likely fail (unless using the latest gnu parallel module, 20130422, which by default puts you in $PWD on the remote node).
* Loaded modules should get automatically loaded on the remote nodes too for the latest gnu parallel module, but not for earlier ones.
* If you need an environment variable to be transfered from the job script to the remotely running subjobs, use <tt>--env ENVIRONMENTVARIABLE</tt>.
Notes:
* Of course, this is just an example of what you could do with gnu parallel. How you set up your specific run depends on how each of the runs would be started. One could for instance also prepare a file of commands to run and make that the input to parallel as well.
* Note that submitting several bunches to single nodes, as in the section above, is a more failsafe way of proceeding, since a node failure would only affect one of these bunches, rather than all runs.
* GNU Parallel can be passed a file with the list of nodes to which to ssh, using <tt>--sshloginfile</tt> (thanks to Ole Tange for pointing this out). This list is automatically generated by the scheduler and its name is made available in the environment variable $PBS_NODEFILE.
* Alternatively, GNU Parallel can take a comma separated list of nodes given to its -S argument, but this would need to be constructed from the file $PBS_NODEFILE which contains all nodes assigned to the job, with each node duplicated 8x for the number of cores on each node.
* GNU Parallel can reads lines of input and convert those to arguments in the execution command. The execution command is the last argument given to <tt>parallel</tt>, with <tt>{}</tt> replaces by the lines on input.
* The --workdir argument is essential: it sets the working directory on the other nodes, which would default to your home directory if omitted. Since /home is read-only on the compute nodes, you would not like not get any output at all! This is no longer true for the latest GNU Parallel modules (20130422), which puts you in the current directory on the remote nodes.
* We reiterate that if memory requirements allow it, you should try to run more than 8 jobs at once, with a maximum of 16 jobs. You can run more or fewer than 8 processes per node by modifying the -j8 parameter to the parallel command.

===More on GNU parallel===
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

===GNU Parallel Reference===
* O. Tange (2011): GNU Parallel - The Command-Line Power Tool, '';login: The USENIX Magazine,'' February 2011:42-47.

===Older scripts===

Older scripts, which mimicked some of GNU parallel functionality, can be found on the [[Deprecated scripts]] page.

--[[User:Rzon|Rzon]] 02:22, 14 Nov 2010 (UTC)

Knowledge Base: Tutorials and Manuals

2014-04-07T21:23:09Z

Ljdursi: /* Programming */

__TOC__
=Training material=

For upcoming classes, see our [https://support.scinet.utoronto.ca/education/ Training and Education website]!
==SciNet Basics==
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* Intro to SciNet: [http://support.scinet.utoronto.ca/CourseVideo/scinetintro/scinetintro.html Video]/[[Media:Introscinet.pdf|Slides]], SciNet, November 2012
* SciNet Resources: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_SciNet_Resources/Monday_Morning_SciNet_Resources.mp4 Video]/ [[Media:Monday_Morning_SciNet_Resources.pdf|Slides]]
* [[Essentials]]
* [[FAQ|Frequenty asked questions]]
* [[Ssh]]
* [[GPC_Quickstart|GPC quickstart]]
* [[TCS_Quickstart|TCS quickstart]]
* [[GPU_Devel_Nodes|ARC/GPU quickstart]]
* [[Cell_Devel_Nodes|ARC/Cell quickstart]]
* [[Important .bashrc guidelines]]
* [[Media:LargeScaleBio.pdf‎|Workflow Optimization (w/focus on Large Scale BioInformatics)]]
* [[Software_and_Libraries | Software and libraries]]
* [[Installing your own modules]]
* [[Media:SNUGlocalsetup.pdf|User-space modules and packages (April 2011 SNUG TechTalk)]]
* [[Media:HPSS_rationale.pdf|HPSS - SciNet's storage capacity expansion]]
* BGQ Hardware Overview [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides ]/ [https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 Video Recording ]
* Intro to Using the BGQ [[Media:Bgqintro.pdf|Slides ]]/[https://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 Video Recording ]

==Linux==
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
* Introduction to the Linux Shell, SciNet, Mar 2012: [[Media:IntroToShell.pdf|Slides]] and [[Media:Shell-data.tgz|Data files]]

==Batch job management==
* [[Media:LargeScaleBio.pdf‎|Workflow Optimization (w/focus on Large Scale BioInformatics)]]
* [[Media:Tech-talk-gnu-parallel.pdf|GNU Parallel (Techtalk Nov 14, 2012)]]
* [[Media:TechTalkJobMonitoring.pdf|Job Monitoring on SciNet and Job Efficiency]]

* [[Wallclock time]]
* [[Checkpoints]]
* [[Using_Signals|Signals]]
* [[Moab]]
* [[User_Serial|Serial Jobs (including GNU Parallel)]]
* [[User_Ramdisk|Ramdisk]]
* [http://www.clusterresources.com/products/mwm/docs/index.shtml Moab workload manager]
* [http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml Moab commands]
* [http://www.clusterresources.com/products/torque/docs/ Torque resource manager]
* [http://www.clusterresources.com/products/torque/docs/a.acommands.shtml Torque PBS commands]
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]

==Programming==
===General===
* [[Media:SciDev-XLCompilers.pdf|Performance Tuning with the IBM XL Compilers]]: Slides fromt the SciNet Develop Seminar by Kit Barton, Sep 17, 2012.
* [[Media:Remotescinet.pdf‎|Remote Development]], slides from TechTalk Jun 13, 2012
* [[Scientific Software Development Course]], part I of the SciNet's Scientific Computing Course
* [http://software-carpentry.org Software Carpentry Resources]
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
* [[IBM_Nov_Workshop | IBM AIX Workshop, SciNet, Nov 2008 ]]
* [[IBM_Compiler_Workshop | IBM Compiler Workshop, SciNet, Feb 2009]]
* SNUG Techtalk Dec 2011 [[Media:Snug_techtalk_compiler.pdf | Intel Compiler Optimizations]]

===Fortran===
* Modern Fortran Course (1 day), SciNet, 19 Apr 2011
** [[Media:ModernFortran.pdf | Slides]]
** [[Media:ModernFortran.tgz | Source Code]]
* [http://software.intel.com/sites/products/documentation/hpc/compilerp* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf IBM Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]

===C++===
* [[Media:Cpp11.pdf|Slides]] and [http://support.scinet.utoronto.ca/CourseVideo/Cpp11/cpp11.html recording] of the SciNet Developer Seminar on C++11, March 20, 2013
* Scientific C++ Course (1 day), SciNet, 15 March 2011
** [[Media:Scientific-c%2B%2B.pdf|Slides]] (updated on Apr 26, 2012)
** [[Media:Scinetcppexamples.tgz|Example source code]]
** [[Videos_of_the_One-Day_Scientific_C%2B%2B_Class | Videos of the Scientific C++ class]]
* [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/cpp/lin/compiler_c/index.htm Intel C & C++ compiler]
ro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran compiler]
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf IBM C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]

===C===
* C refresher: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Monday_Morning_C_Review/Monday_Morning_C_Review.mp4 Video]/ [[Media:Monday_Morning_C_Review.pdf| Slides]]
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf IBM C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]

===Hadoop===
* Introduction to Hadoop for HPCers, Part I - MapReduce: [[Media:Hadoop-PartI.pdf|Slides]], [[Media:HadoopPart1examples.tgz|Source Code]], [http://support.scinet.utoronto.ca/~ljdursi/SciNetHadoopVM.zip Virtual Machine]

===Perl===
* [[Perl]]
===Python===
* [[Python]]
* [[IPython Notebook on GPC]] (January 2014 TechTalk)
* [[Research Computing with Python]] (Modular Course, Fall 2013)
* [http://support.scinet.utoronto.ca/Snug/scinet-f2py/scinet-f2py.html f2py: Fortran and Python] (June 2011 TechTalk by Pierre de Buyl)

===R===
* [[R Statistical Package]]
===Lua===
* [[Media:PeterColberg_Lua_scinet.pdf | Scripting HALMD with Lua and Luabind]] (May 2011 TechTalk by Peter Colberg)

==Parallel Programming==
* [[Ontario Summerschool on High Performance Computing Central]]
* [[High Performance Scientific Computing]], part 3 of SciNet's Scientific Computing Course (Winter 2012)
* Parallel Programming Course (5 days), SciNet, May 2011
** [[Parallel_Scientific_Computing_-_May_2011 | Videos, slides and code]]
* Parallel Computing for Computational Fluid Dynamics (CFD), SciNet, 23 March 2011
** [[Media:parCFD-mpi.pdf | Slides]]
** [[Media:parCFD.tgz | Source Code]]
* Intro to Practical Parallel Programming (1 day), SciNet, 22 Sept 2010:
**[[Media:PPP-Intro-Morning.pdf|Morning Slides, Intro and OpenMP ]]
**[[Media:PPP-Intro-Afternoon.pdf|Afternoon Slides, MPI]]
**[[Media:Intro-ppp.tgz|Example source code]]
* Parallel Scientific Computing Workshop (5 days), SciNet, Aug 2009:
**[[ Parallel_Scientific_Computing_-_Aug_09 | Slides ]]
**[http://www.cita.utoronto.ca/~ljdursi/PSP/ Video]
* [http://www.vscse.org/ Virtual School for CSE] Web courses (Jul/Aug 2010):
** Petascale programming environments and tools
** Big data for science
** Proven algorithmic techniques for many-core processors
* [https://computing.llnl.gov/tutorials/mpi/ LLNL MPI Tutorial]: This was the basis for the MPI workshop at SciNet.
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]
* [[GPC MPI Versions]]
* [[Co-array Fortran on the GPC]]
* [[IBM_Feb_Workshop | IBM MPI Workshop, SciNet, Feb 2009]]
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf IBM UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]

==GPU Computing==

* [[Media:Westgrid_CUDA.pdf | Intro to GPU Computing Using CUDA]] (WestGrid Spring 2014 Seminar Series)
* 1.5 hour intro to CUDA, March 2013: [[Media:CUDA-Graphics-Intro-2013.pdf | Slides]] and [[Media:CUDA-Graphics-Intro-2013.tgz | Source Code]]
* [[CUDA_Minicourse_Fall_2012 | CITA/SciNet CUDA Minicourse, Fall 2012]]
* [[SciNet GPU Workshop July 2010]]
* Intro to GPGPU Programming: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_GPGPU/Friday_Morning_GPGPU.mp4 Video]/ [[Media:Gpgpu.pdf | Slides]] (from 5 day parallel programming course at SciNet, May 2011)
* 1-day intro to GPGPU using CUDA Course (Aug 2011): [[Media:Intro-gpu.tgz | Source Code]], [[Media:IntroGPGPU-Aug2011.pdf | Slides]].
* [http://developer.nvidia.com/object/cuda_training.html NVidia archived courses for GPGPU Programming]
* [http://www.pgroup.com/doc/pgiug.pdf PGI Compiler User's Guide]
* [http://www.pgroup.com/doc/pgiref.pdf PGI Compiler Reference Manual]
* [http://www.pgroup.com/doc/pgifortref.pdf PGI Fortran reference]
* [http://www.pgroup.com/doc/pgicudaforug.pdf PGI CUDA Fortran Programming Guide and Reference]
* [http://www.pgroup.com/doc/openACC_gs.pdf PGI OpenACC Getting Started Guide]

==Performance Tuning==
* [[Performance and Profiling Course, April 2013]]
* [[Introduction To Performance]]
* Performance tools for [[Performance_And_Debugging_Tools:_GPC | GPC ]] and [[Performance_And_Debugging_Tools:_TCS | TCS ]]
* Dec 2010 SNUG TechTalk: [[Media:ProfillingTechTalk-Dec2010.pdf | Profiling Tools on GPC]]
* [http://cnx.org/content/col11136/latest/ High Performance Computing Book] Online version of an older O'Reilly book which covers the basics of (mostly serial) programming for performance. Covers the most important issues today very clearly.
* [http://www.ece.cmu.edu/~franzf/papers/gttse07.pdf How to Write Fast Numerical Code ] Good introduction to thinking about performance.
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]
* [[Media:Mpi-tuning-parameters.pdf‎ | MPI Tuning Parameters]] - SNUG TechTalk, Feb 2012

==Debugging==
* [[Media:SS_Debug.pdf|Debugging with GDB and DDT, half-day session at the Ontario HPC Summerschool 2012 Central Slides]], [[Media:SS_Debug.tgz|Code]].
* [[Media:Snugdebug.pdf|TechTalk: Debuggers & Parallel Debugging on SciNet - gdb, ddd, padb], SciNet User Group Meeting, Nov 2010]] [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_Debugging/Thursday_Morning_Debugging.mp4 Video]
* [http://www.allinea.com/downloads/userguide.pdf Allinea DDT (Distributed Debugging Tool) User Guide]

==Math libraries (BLAS, LAPACK, FFT)==
* [[Media:MKLTechTalkMarch2012.pdf|Intel Math Kernel Library (MKL): An overview]] (TechTalk, March, 2012)
* [[Numerical Tools for Physical Scientists]], part 2 of SciNet's Scientific computing course, covers, random nubers, blas, lapack, fft, ...
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)]
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor] ($MKLPATH → ${MKLPATH} in makefiles)
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])
* [http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf Parallel ESSL high performance math library V3.3]
* [http://hal.inria.fr/inria-00576469 Linear Algebra Libraries] by Claire Mouton. 2009 INRIA Technical Report on existing linear algebra libraries for C++ (also here: [http://arxiv.org/abs/1103.3020])

==I/O==

* [[Media:NetCDF.pdf|Introduction to NetCDF4 binary files with Python, C++ and R (TechTalk March 2014)]]
* [[Media:SCIENCEDATA.pdf‎|Sep 2012 SNUG TechTalk: Science=Data]]
* [[Data_Management|Data management]]
* Intro to Parallel I/O, SciNet, Oct 6th, 2010:
**[[Media:Parallel_io_course.pdf‎|Morning & MPI-IO Slides]]
**[[Media:Netcdfhdf5.pdf|NetCDF/HDF5 Slides]]
**[[Media:ParIO.tgz|Source Code]].
* Half-day HPCS2012 Parallel I/O tutorial, covering MPI-IO, HDF5, NetCDF, based on the above: [[Media:ParIO-HPCS2012.pdf|slides (pdf)]] and [[Media:ParIO-HPCS2012.tgz|source code]].
* [[Media:Snugio.pdf|Sept 2010 SNUG TechTalk: Parallel File System and IO]] [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Friday_Morning_IO/Friday_Morning_IO.mp4 Video]
* [[File System and I/O dos and don'ts]]
* [[Media:40TB.pdf|So you have 40TB of Data]] -- an overview of things to consider with large data sets.
* [[Media:Adios-techtalk-may2012.pdf|May 2012 SNUG TechTalk: ADIOS for Parallel IO slides]] and [[Media:Adios-techtalk-may2012-src.tgz|source code]]
* [[hdf5_table|Writting / Reading a table in hdf5]]
* [[NetCDF_table|Writting / Reading a table in NetCDF]]

==Infiniband Networking==
* [[Media:Snug_techtalk_Infiniband.pdf | TechTalk on SciNet's Infiniband Network & MPI options ]]

==Visualization==
* [[Using Paraview]]
* [[Media:Ttvnc.pdf|TechTalk on VNC (slides)]]
* [[Software_and_Libraries#anchor_viz|Visualization Software on the GPC]]
* [http://scienceillustrated.ca Science Illustrated:] Two-day symposium on Visualizing Science, Feb 2011
* [http://www.kmdi.utoronto.ca/story/2011/03/si-science-illustrated-symposium-success Videos of the talks given at Science Illustrated] (recorded by [http://www.kmdi.utoronto.ca KMDI] at [http://www.utoronto.ca UoT]):
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=94ff5cd5-be6e-4fc6-9be6-dd2222342bcd Opening remarks] by Paul Young
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4255c34e-15e7-4b24-ba99-78f5c8fa4381 Information Visualization and the Myth of Information Overload] by Christopher Collins
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=adcf02bf-16cb-46cc-8cdc-1a65e9071d6b Beyond Basic Visualization] by Ramses Van Zon
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=47baf346-3599-4fa3-9b10-6a58faa6b33c Network Visualization & Analysis] by Igor Jurisica
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7e754a2e-7be5-476e-bb54-37def37bc07e Simulation and Visualization of Blood Flow] by David Steinman
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=d7622587-2c31-49c1-99d7-f0f16c078801 Scientific Visualizations: Does the Science Matter?] by Thomas Lucas
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=9963c637-6840-454f-a57f-9a1be6456616 How can visualization impact public perception of science?] Panelists: Jay Ingram, Peter Calamai, Reni Barlow, Hooley McLaughlin
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=88e50cff-db9b-4c71-b10d-781fec60a2c0 How Info Graphics are Created for the Mainstream Media] by Peter Calamai
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=3897e2a3-1fda-42be-ab78-edbab090fd9e Design Boot Camp] by Graham Huber
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=4c27ed76-7292-407e-83a6-814e1461eccd Visualization Large Datasets] by Jonathan Dursi
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d49c845-3937-44e2-a300-9b8ffe57a857 Visualizing Colliding Black Holes] by Herald Pfeiffer
** [http://itube.ischool.utoronto.ca/Panopto/Pages/Viewer/Default.aspx?id=7d7e4803-39cd-4c8e-a443-bc2a7b1b3c28 Closing remarks] by Mubdi Rahman

==Applications==
{{:Knowledge Base: Applications}}
* See also [[User Codes]]

=Manuals=

==Intel compilers and libraries (GPC)==
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/cpp-lin/index.htm C & C++ compiler]
* [http://software.intel.com/sites/products/documentation/doclib/stdxe/2013/composerxe/compiler/fortran-lin/index.htm Fortran compiler]
* [[Media:FP_Consistency.pdf|Intel Compiler Floating Point Consistency]]
* [[Media:Compiler_qrg12.pdf‎|Intel Compiler Optimization Guide]]
* [http://software.intel.com/sites/products/documentation/hpc/mkl/lin/index.htm Math Kernel Library (MKL)]
* [http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vslnotes.pdf Math Kernel Library's Vector Statistical Library]
* [http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf Intel MPI library]
* [http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor Math Kernel Library link line advisor] ($MKLPATH → ${MKLPATH} in makefiles)

==IBM compilers and libraries (TCS/P7)==
* [http://support.scinet.utoronto.ca/Manuals/xlc-compiler.pdf C compiler], [http://support.scinet.utoronto.ca/Manuals/xlc-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlc-proguide.pdf optimization]
* [http://support.scinet.utoronto.ca/Manuals/xlC++-compiler.pdf C++ compiler], [http://support.scinet.utoronto.ca/Manuals/xlC++-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlC++-proguide.pdf optimization]
* [http://support.scinet.utoronto.ca/Manuals/xlf-compiler.pdf Fortran compiler] [http://support.scinet.utoronto.ca/Manuals/xlf-langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/xlf-proguide.pdf optimization]
* [http://support.scinet.utoronto.ca/Manuals/UPC/compiler.pdf UPC compiler], [http://support.scinet.utoronto.ca/Manuals/UPC/langref.pdf language], [http://support.scinet.utoronto.ca/Manuals/UPC/upcopt.pdf optimization], [http://support.scinet.utoronto.ca/Manuals/UPC/standlib.pdf library], [http://support.scinet.utoronto.ca/Manuals/UPC/upcusersguide.pdf user's guide], [http://support.scinet.utoronto.ca/Manuals/UPC/proguide.pdf programmer's guide]
* [http://publib.boulder.ibm.com/epubs/pdf/am501405.pdf ESSL high performance math library V4] ([http://publib.boulder.ibm.com/epubs/pdf/am601305.pdf V3])
* [[Media:essl51.pdf|ESSL high performance math library V5.1 for Linux on Power]]
* [http://support.scinet.utoronto.ca/Manuals/JUMP-AIX-POWER6-AppsPerformanceTuning-wp032008.pdf Performance tuning]
* [http://support.scinet.utoronto.ca/Manuals/PE5.1-operationanduse.pdf Parallel environment]
* [http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp Cluster information center] (with error codes)
* [http://support.scinet.utoronto.ca/Manuals/LL-usingandadministering.pdf LoadLeveler: using & administering]

==PGI compilers (ARC)==
* [http://www.pgroup.com/doc/pgiug.pdf Compiler User's Guide]
* [http://www.pgroup.com/doc/pgiref.pdf Compiler Reference Manual]
* [http://www.pgroup.com/doc/pgifortref.pdf Fortran reference]
* [http://www.pgroup.com/doc/pgicudaforug.pdf CUDA Fortran Programming Guide and Reference]
* [http://www.pgroup.com/doc/openACC_gs.pdf OpenACC Getting Started Guide] (Note: $PGI/linux86-64/12.5/doc contains a newer version.)

==Scheduler (Adaptive Computing/Cluster Resources)==
* [http://www.clusterresources.com/products/mwm/docs/index.shtml Moab workload manager]
* [http://www.clusterresources.com/products/mwm/docs/a.gcommandoverview.shtml Moab commands]
* [http://www.clusterresources.com/products/torque/docs/ Torque resource manager]
* [http://www.clusterresources.com/products/torque/docs/a.acommands.shtml Torque PBS commands]

==DDT Debugger (Allinea)==
* [http://www.allinea.com/downloads/userguide.pdf Distributed Debugging Tool User Guide]

File:HadoopPart1examples.tgz

2014-04-07T21:19:03Z

Ljdursi: Source code examples for Hadoop part 1

Source code examples for Hadoop part 1

File:Hadoop-PartI.pdf

2014-04-07T21:18:05Z

Ljdursi: Slides for Introduction to Hadoop for HPCers, Part I

Slides for Introduction to Hadoop for HPCers, Part I

Ssh keys

2013-07-16T17:14:59Z

Ljdursi: /* SSH tunnel */

[[Ssh | SSH]] has an alternative to passwords to authenticate your login; you can generate a key file on a trusted machine and tell a remote machine to trust logins from a machine that presents that key. This can be both convenient and secure, and may be necessary for some tasks (such as connecting directly to compute nodes to use [[Using_Paraview | some visualization packages]]). Here we describe how to setup keys for logging into SciNet.

==SSH Keys and SciNet==

[[Ssh | SSH]] is a secure protocol for logging into or copying data to/from remote machines. In addition to using passwords to [http://en.wikipedia.org/wiki/Authentication authenticate] users, one can use cryptographically secure keys to guarantee that a login request is coming from a trusted account on a remote machine, and automatically allow such requests. Done properly, this is as secure as requiring a password, but can be more convenient, and is necessary for some operations.

On this page, we will assume you are using Linux, Mac OS X, or a similar environment such as [http://www.cygwin.com/ Cygwin] under Windows. If not, the steps will be the same, but how they are done (for instance, generating keys) may differ; look up the documentation for your ssh package for details.

==Using SSH keys==
===How SSH keys work===

SSH relies on [http://en.wikipedia.org/wiki/Public-key_cryptography public key cryptography] for its encryption. These cryptosystems have a private key, which must be kept secret, and a public key, which may be disseminated freely. In these systems, anyone may use the public key to encode a message; but only the owner of the private key can decode the message. This can also be used to verify identities; if someone is claiming to be Alice, the owner of some private key, Bob can send Alice a message encoded with Alice's well-known public key. If the person claiming to be Alice can then tell Bob what the message really was, then that person at the very least has access to Alice's private key.

To use keys for authentication, we:
* Generate a key pair (Private and Public)
* Copy the public keys to remote sites we wish to be able to login to, and mark it as an authorized key for that system
* Ensure permissions are set properly
* Test.

===Generating an SSH key pair===

{|border="1"
|
Note: This describes creating ssh key pairs on '''your''' machine, not on SciNet. On SciNet, you already have key pairs generated, sitting in <tt>${HOME}/.ssh/</tt>, and modifying them is likely to cause problems.
|}

The first stage is to create an SSH key pair. On most systems, this is done using the command

ssh-keygen

This will prompt you for two pieces of information: where to save the key, and a passphrase for the key. The passphrase is like a password, but rather than letting you in to some particular account, it allows you to use the key you've generated to log into other systems.

There are a series of possible options to <tt>ssh-keygen</tt> which allow increasingly cryptographically secure keys (by increasing the number of bits used in the key), or by choosing different encryption systems. The defaults are fine, and we won't discuss other options here.

The default location to save the private key is in <tt>${HOME}/.ssh/id_rsa</tt> (for an RSA key); unless you have some specific reason for placing it elsewhere, use this option. The public key will be <tt>id_rsa.pub</tt> in the same directory.

Your passphrase can be any string, and of any length. It is best not to make it the same as any of your passwords.

A sample session of generating a key would go like this:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (${HOME}/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ${HOME}/.ssh/id_rsa.
Your public key has been saved in ${HOME}/.ssh/id_rsa.pub.
The key fingerprint is:
79:8e:36:6a:78:7d:cf:80:94:90:92:0e:74:0b:f1:b7 USERNAME@YOURMACHINE

====Don't Use Passphraseless Keys!====

If you do not specify a passphrase, you will have a completely "exposed" private key. '''This is a terrible idea.''' If you then use this key for anything it means that anyone who sits down at your desk, or anyone who borrows or steals your laptop, can login to anywhere you use that key (good guesses could come from just looking at your history) without needing any password, and could do anything they wanted with your account or data. Don't use passphraseless keys.

We should note that we do, in fact, have one necessary and reasonable exception here -- the keys used within SciNet itself. The SciNet key used for within-scinet operations (you already have one in your account in <tt>~/.ssh/id_rsa</tt>) is passphraseless, for two good reasons. One is that, once you are on one SciNet machine (like the login node), you already have read/write access to all your data; all the nodes mount the same file systems. So there is little to be gained in protecting the SciNet nodes from each other. The second is practical; ssh is used to login to compute nodes to start your compute jobs. You obviously can't be asked to type in a passphrase every time one of your jobs starts; you may not be at your computer at that moment. So passphraseless keys are ok ''within'' a controlled environment; but don't use them for remote access.

===Copying the Public Key to SciNet (and elsewhere)===

Now that you have this SSH "identity", you use the public (''not'' the private) key for access to remote machines. The public key must be put as one line in the file <tt>/home/USERNAME/.ssh/authorized_keys</tt>. Do not delete the lines already there, or you may end up with strange problems using SciNet machines.

You can copy your new public key to the SciNet systems
<pre>
scp /home/LOCAL_USERNAME/.ssh/id_rsa.pub SCINET_USERNAME@login.scinet.utoronto.ca:newkey
</pre>
Then login to SciNet and copy&paste the contents from <tt>~/newkey</tt> into <tt>~/.ssh/authorized_keys</tt>.
<pre>
cat ~/newkey >> ~/.ssh/authorized_keys
</pre>

===<tt>.ssh</tt> Permissions===

Note that <tt>SSH</tt> is very fussy about file permissions; your <tt>~/.ssh</tt> directory must only be accessible by you, and your various key files must not be writable (or in some cases, readable) by anyone else. Sometimes users accidentally reset file permissions while editing these files, and problems happen. If you look at the <tt>~/.ssh</tt> directory itself, it should not be readable at all by anyone else:

ls -ld ~/.ssh
drwx------ 2 USERNAME GROUPNAME 7 Aug 9 15:43 /home/USERNAME/.ssh

and <tt>authorized_keys</tt> must not be writable:

$ ls -l ~/.ssh/authorized_keys
-rw-r--r-- 1 USERNAME GROUPNAME 1213 May 29 2009 /home/USERNAME/.ssh/authorized_keys

===Testing Your Key===

Now you should be able to login to the remote system (say, SciNet):

$ ssh USERNAME@login.scinet.utoronto.ca
Enter passphrase for key '/home/USERNAME/.ssh/id_rsa':
Last login: Tue Aug 17 11:24:48 2010 from HOMEMACHINE

===================================================

This SciNet login node is to be used only as a
gateway to the GPC and TCS.

[...]
scinet04-$

If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.
Please contact your system administrator.
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of
this message.
Offending key in /home/<user>/.ssh/known_hosts:3
RSA host key for login.scinet.utoronto.ca
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has
changed and you have requested
</pre>

* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be
restarted with the new passphrase ssh key.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.
</pre>

===(Optional) Using <tt>ssh-agent</tt> to Remember Your Passphrase===

But now you've just replaced having to type a password for login with having to type a passphrase for your key; what have you gained?

It turns out that there's an automated way to manage ssh "identities", using the <tt>ssh-agent</tt> command, which should automatically be running on newer Linux or Mac OS X machines. You can add keys to this agent for the duration of your login using the <tt>ssh-add</tt> command:

$ ssh-add
Enter passphrase for /home/USERNAME/.ssh/id_rsa:
Identity added: /home/USERNAME/.ssh/id_rsa (/home/USERNAME/.ssh/id_rsa)

and then logins will not require the passphrase, as <tt>ssh-agent</tt> will provide access to the key.

When you log out of your home computer, the ssh agent will close, and next time you log in, you will have to <tt>ssh-add</tt> your key. You can also set a timeout of (say) an hour by using <tt>ssh-add -t 3600</tt>. This minimizes the number of times you have to type your passphrase, while still maintaining some degree of key security.

=== Multiple ssh private keys ===
In quite a few situations its preferred to have ssh keys dedicated to each service, specific role or domain.
<pre>
ssh-keygen -t rsa -f ~/.ssh/id_rsa.SciNet -C "Key for SciNet"
ssh-keygen -t rsa -f ~/.ssh/id_rsa.SHARCNET -C "Key for SHARCNET"
ssh-keygen -t rsa -f ~/.ssh/id_rsa.DCS -C "Key for Dept. Of Computer Science"
ssh-keygen -t rsa -f ~/.ssh/id_rsa.CITA -C "Key for CITA"
</pre>

Use different file names for each key. Lets assume that there are 2 keys, ~/.ssh/id_rsa.SciNet and ~/.ssh/id_rsa.SHARCNET. The simple way of making sure each of the keys works all the time is to now create config file for ssh:
<pre>
touch ~/.ssh/config
chmod 600 ~/.ssh/config
echo "IdentityFile ~/.ssh/id_rsa.SciNet" >> ~/.ssh/config
echo "IdentityFile ~/.ssh/id_rsa.SHARCNET" >> ~/.ssh/config
</pre>

This would make sure that both keys are always used whenever ssh makes a connection. However, ssh config lets you get down to a much finer level of control on keys and other per-connection setups. The recommendation is to use a key selection based on the Hostname. For example, a ~/.ssh/config that looks like this :

<pre>
Host SciNet
Hostname login.scinet.utoronto.ca
IdentityFile ~/.ssh/id_dsa.SciNet
User pinto

Host SHARCNET
Hostname sharcnet.ca
IdentityFile ~/.ssh/id_rsa.SHARCNET
User jchong
Port 44787
</pre>

And just login with the shortcut:
<pre>
$ ssh SciNet
</pre>

= SSH tunnel =
A more obscure use of ssh is to generate a communication tunnel. As an example, assume you want to access a website running on a remotehost using your localhost, but there is a firewall between the 2 systems blocking every port, except incoming ssh.

The basic syntax of the ssh command for such a purpose is:
<pre>
ssh -f -N -L localport:localhost:remoteport user@remotehost
# -f puts ssh in background
# -N makes it not execute a remote command
</pre>

If the remote website broadcasts on the default port 80, you could do the following:
<pre>
ssh -L 8080:localhost:remotehost:80 tunneluser@remotehost
</pre>
... and point your local browser to http://localhost:8080

If you don't want to remember the above sequence of flags all the time, you can add an entry to your ~/.ssh/config:
<pre>
Host tunnel
HostName remotehost
IdentityFile ~/.ssh/id_rsa.tunnel
LocalForward 8080 127.0.0.1:80
User tunneluser
</pre>

To open the tunnel just issue the command:
<pre>
$ ssh -f -N tunnel
</pre>

FAQ

2013-07-16T17:07:34Z

Ljdursi: /* How do we manage job priorities within our research group? */

__TOC__

==The Basics==
===Whom do I contact for support?===

Whom do I contact if I have problems or questions about how to use the SciNet systems?

'''Answer:'''

E-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

In your email, please include the following information:

* your username on SciNet
* the cluster that your question pertains to (GPC or TCS; SciNet is not a cluster!),
* any relevant error messages
* the commands you typed before the errors occured
* the path to your code (if applicable)
* the location of the job scripts (if applicable)
* the directory from which it was submitted (if applicable)
* a description of what it is supposed to do (if applicable)
* if your problem is about connecting to SciNet, the type of computer you are connecting from.

Note that your password should never, never, never be to sent to us, even if your question is about your account.

Try to avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team!

===What does ''code scaling'' mean?===

'''Answer:'''

Please see [[Introduction_To_Performance#Parallel_Speedup|A Performance Primer]]

===What do you mean by ''throughput''?===

'''Answer:'''

Please see [[Introduction_To_Performance#Throughput|A Performance Primer]].

Here is a simple example:

Suppose you need to do 10 computations. Say each of these runs for
1 day on 8 cores, but they take "only" 18 hours on 16 cores. What is the
fastest way to get all 10 computations done - as 8-core jobs or as
16-core jobs? Let us assume you have 2 nodes at your disposal.
The answer, after some simple arithmetic, is that running your 10
jobs as 8-core jobs will take 5 days, whereas if you ran them
as 16-core jobs it would take 7.5 days. Take your own conclusions...

===I changed my .bashrc/.bash_profile and now nothing works===

The default startup scripts provided by SciNet, and guidelines for them, can be found [[Important_.bashrc_guidelines|here]]. Certain things - like sourcing <tt>/etc/profile</tt>
and <tt>/etc/bashrc</tt> are ''required'' for various SciNet routines to work!

If the situation is so bad that you cannot even log in, please send email [mailto:support@scinet.utoronto.ca support].

===Could I have my login shell changed to (t)csh?===

The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" <tt>#!/bin/tcsh</tt> at the top.

===How can I run Matlab / IDL / Gaussian / my favourite commercial software at SciNet?===

'''Answer:'''

Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).

If your research group requires a commercial package that you already have or are willing to buy licenses for, contact us at [mailto:support@scinet.utoronto.ca support@scinet] and we can work together to find out if it is feasible to implement the packages licensing arrangement on the SciNet clusters, and if so, what is the the best way to do it.

Note that it is important that you contact us before installing commercially licensed software on SciNet machines, even if you have a way to do it in your own directory without requiring sysadmin intervention. It puts us in a very awkward position if someone is found to be running unlicensed or invalidly licensed software on our systems, so we need to be aware of what is being installed where.

===Do you have a recommended ssh program that will allow scinet access from Windows machines?===

'''Answer:'''

The [[Ssh#SSH_for_Windows_Users | SSH for Windows users]] programs we recommend are:

* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools, including ssh and X, all wrapped up into one executable.
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] - this is a terminal for windows that connects via ssh. It is a quick install and will get you up and running quickly. To set up your passphrase protected ssh key with putty, see [http://the.earth.li/~sgtatham/putty/0.61/htmldoc/Chapter8.html#pubkey here].
* [http://www.cygwin.com/ CygWin] - this is a whole linux-like environment for windows, which also includes an X window server so that you can display remote windows on your desktop. Make sure you include the openssh and X window system in the installation for full functionality. This is recommended if you will be doing a lot of work on Linux machines, as it makes a very similar environment available on your computer. To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.
 To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.

===My ssh key does not work! WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ===

'''Answer:'''

[[Ssh_keys#Testing_Your_Key | Testing Your Key]]

* If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.
Please contact your system administrator.
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of
this message.
Offending key in /home/<user>/.ssh/known_hosts:3
RSA host key for login.scinet.utoronto.ca
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has
changed and you have requested
</pre>

* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be
restarted with the new passphrase ssh key.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.
</pre>

===Can't forward X: "Warning: No xauth data; using fake authentication data", or "X11 connection rejected because of wrong authentication."===

I used to be able to forward X11 windows from SciNet to my home machine, but now I'm getting these messages; what's wrong?

'''Answer:'''

This very likely means that ssh/xauth can't update your ${HOME}/.Xauthority file.

The simplest pssible reason for this is that you've filled your 10GB /home quota and so can't write anything to your home directory. Use

<pre>
$ module load extras
$ diskUsage
</pre>

to check to see how close you are to your disk usage on ${HOME}.

Alternately, this could mean your .Xauthority file has become broken/corrupted/confused some how, in which case you can delete that file, and when you next log in you'll get a similar warning message involving creating .Xauthority, but things should work.

===How come I can not login to TCS?===

'''Answer:'''

A SciNet account doesn't automatically entitle you to TCS access. At a minimum, TCS jobs need to run on at least 32 cores (64 preferred because of Simultaneous Multi Threading - [[TCS_Quickstart#Node_configuration|SMT]] - on these nodes) and need the large memory (4GB/core) and bandwidth on the system. Essentially you need to be able to explain why the work can't be done on the GPC.

===How can I reset the password for my Compute Canada account?===

'''Answer:'''

You can reset your password for your Compute Canada account here:

https://ccdb.computecanada.org/security/forgot

===How can I change or reset the password for my SciNet account?===

'''Answer:'''

To reset your password at SciNet please e-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

If you know your old password and want to change it, that can be done here:

https://portal.scinet.utoronto.ca/

===Why am I getting the error "Permission denied (publickey,gssapi-with-mic,password)"?===

This error can pop up in a variety of situations: when trying to log in, or when after a job has finished, when the error and output files fail to be copied (there are other possible reasons for this failure as well -- see [[FAQ#My_GPC_job_died.2C_telling_me_.60Copy_Stageout_Files_Failed.27|My GPC job died, telling me:Copy Stageout Files Failed]]).
In most cases, the "Permission denioed" error is caused by incorrect permission of the (hidden) .ssh directory. Ssh is used for logging in as well as for the copying of the standard error and output files after a job.

For security reasons,
the directory .ssh should only be writable and readable to you, but yours
has read permission for everybody, and thus it fails. You can change
this by
<pre>
chmod 700 ~/.ssh
</pre>
And to be sure, also do
<pre>
chmod 600 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub ~/authorized_keys
</pre>

===ERROR:102: Tcl command execution failed? when loading modules ===
Modules sometimes require other modules to be loaded first.
Module will let you know if you didn’t.
For example:
<pre>
$ module purge
$ module load python
python/2.6.2(11):ERROR:151: Module ’python/2.6.2’ depends on one of the module(s) ’gcc/4.4.0’
python/2.6.2(11):ERROR:102: Tcl command execution failed: prereq gcc/4.4.0
$ gpc-f103n084-$ module load gcc python
$
</pre>

==Compiling your Code==

===How can I get g77 to work?===

The fortran 77 compilers on the GPC are ifort and gfortran. We have dropped support for g77. This has been a conscious decision. g77 (and the associated library libg2c) were completely replaced six years ago (Apr 2005) by the gcc 4.x branch, and haven't undergone any updates at all, even bug fixes, for over five years.
If we would install g77 and libg2c, we would have to deal with the inevitable confusion caused when users accidentally link against the old, broken, wrong versions of the gcc libraries instead of the correct current versions.

If your code for some reason specifically requires five-plus-year-old libraries, availability, compatibility, and unfixed-known-bug problems are only going to get worse for you over time, and this might be as good an opportunity as any to address those issues.

''A note on porting to gfortran or ifort:''

While gfortran and ifort are rather compatible with g77, one
important difference is that by default, gfortran does not preserve
local variables between function calls, while g77 does. Preserved
local variables are for instance often used in implementations of quasi-random number
generators. Proper fortran requires to declare such variables as SAVE
but not all old code does this.
Luckily, you can change gfortran's default behavior with the flag
<tt>-fno-automatic</tt>. For ifort, the corresponding flag is <tt>-noautomatic</tt>.

===Where is libg2c.so?===

libg2c.so is part of the g77 compiler, for which we dropped support. See [[#How can I get g77 to work on the GPC?]] for our reasons.

===Autoparallelization does not work!===

I compiled my code with the <tt>-qsmp=omp,auto</tt> option, and then I specified that it should be run with 64 threads - with
export OMP_NUM_THREADS=64

However, when I check the load using <tt>llq1 -n</tt>, it shows a load on the node of 1.37. Why?

'''Answer:'''

Using the autoparallelization will only get you so far. In fact, it usually does not do too much. What is helpful is to run the compiler with the <tt>-qreport</tt> option, and then read the output listing carefully to see where the compiler thought it could parallelize, where it could not, and the reasons for this. Then you can go back to your code and carefully try to address each of the issues brought up by the compiler.
We ''emphasize'' that this is just a rough first guide, and that the compilers are still not magical! For more sophisticated approaches to parallelizing your code, email us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] to set up an appointment with one
of our technical analysts.

===How do I link against the Intel Math Kernel Library?===

If you need to link in the Intel Math Kernel Library (MKL) libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code.

'''''Note that this give the link line for the command line. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''''

'''''Note too that, unless the integer arguments you will be passing to the MKL libraries are actually 64-bit integers, rather than the normal int or INTEGER types, you want to specify 32-bit integers (lp64) .'''''

===Can the compilers on the login nodes be disabled to prevent accidentally using them?===

'''Answer:'''

You can accomplish this by modifying your .bashrc to not load the compiler modules. See [[Important .bashrc guidelines]].

==="relocation truncated to fit: R_X86_64_PC32": Huh?===

What does this mean, and why can't I compile this code?

'''Answer:'''

Welcome to the joys of the x86 architecture! You're probably having trouble building arrays larger than 2GB, individually or together. Generally, you have to try to use the medium or large x86 `memory model'. For the intel compilers, this is specified with the compile options

-mcmodel=medium -shared-intel

==="feupdateenv is not implemented and will always fail"===

How do I get rid of this and what does it mean?

'''Answer:'''
First note that, as ominous as it sounds, this is really just a warning, and has to do with the intel math library. You can ignore it (unless you really are trying to manually change the exception handlers for floating point exceptions such as divide by zero), or take the safe road and get rid off it by linking with the intel math functions library:<pre>-limf</pre>See also [[#How do I link against the Intel Math Kernel Library?]]

===Cannot find rdmacm library when compiling on GPC===

I get the following error building my code on GPC: "<tt>ld: cannot find -lrdmacm</tt>". Where can I find this library?

'''Answer:'''

This library is part of the MPI libraries; if your compiler is having problems picking it up, it probably means you are mistakenly trying to compile on the login nodes (scinet01..scinet04). The login nodes aren't part of the GPC; they are for logging into the data centre only. From there you must go to the GPC or TCS development nodes to do any real work.

=== Why do I get this error when I try to compile: "icpc: error #10001: could not find directory in which /usr/bin/g++41 resides" ?===

You are trying to compile on the login nodes. As described in the wiki ( https://support.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Login ), or in the users guide you would have received with your account, Scinet supports two main clusters, with very different architectures. Compilation must be done on the development nodes of the appropriate cluster (in this case, gpc01-04). Thus, log into gpc01, gpc02, gpc03, or gpc04, and compile from there.

==Testing your Code==

=== Can I run a something for a short time on the development nodes? ===

I am in the process of playing around with the mpi calls in my code to get it to work. I do a lot of tests and each of them takes a couple of seconds only. Can I do this on the development nodes?

'''Answer:'''

Yes, as long as it's very brief (a few minutes). People use the development nodes
for their work, and you don't want to bog it down for people, and testing a real
code can chew up a lot more resources than compiling, etc. The procedures differ
depending on what machine you're using.

==== TCS ====

On the TCS you can run small MPI jobs on the tcs02 node, which is meant for
development use. But even for this test run on one node, you'll need a host file --
a list of hosts (in this case, all tcs-f11n06, which is the `real' name of tcs02)
that the job will run on. Create a file called `hostfile' containing the following:

tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06

for a 4-task run. When you invoke "poe" or "mpirun", there are runtime
arguments that you specify pointing to this file. You can also specify it
in an environment variable MP_HOSTFILE, so, if your file is in your /scratch directory, say
${SCRATCH}/hostfile, then you would do

<pre>
export MP_HOSTFILE=${SCRATCH}/hostfile
</pre>

in your shell. You will also need to create a <tt>.rhosts</tt> file in your
home director, again listing <tt>tcs-f11n06</tt> so that <tt>poe</tt>
can start jobs. After that you can simply run your program. You can use
mpiexec:

<pre>
mpiexec -n 4 my_test_program
</pre>

adding <tt> -hostfile /path/to/my/hostfile</tt> if you did not set the environment
variable above. Alternatively, you can run it with the poe command (do a "man poe" for details), or even by
just directly running it. In this case the number of MPI processes will by default
be the number of entries in your hostfile.

==== GPC ====

On the GPC one can run short test jobs on the GPC [[GPC_Quickstart#Compile.2FDevel_Nodes | development nodes ]]<tt>gpc01</tt>..<tt>gpc04</tt>;
if they are single-node jobs (which they should be) they don't need a hostfile. Even better, though, is to request an [[ Moab#Interactive | interactive ]] job and run the tests either in regular batch queue or using a short high availability [[ Moab#debug | debug ]] queue that is reserved for this purpose.

=== How do I run a longer (but still shorter than an hour) test job quickly ? ===

'''Answer'''

On the GPC there is a high turnover short queue called [[ Moab#debug | debug ]] that is designed for
this purpose. You can use it by adding
<pre>
#PBS -q debug
</pre>
to your submission script.

==Running your jobs==

===My job can't write to /home===

My code works fine when I test on the development nodes, but when I submit a job, or even run interactively in the development queue on GPC, it fails. What's wrong?

'''Answer:'''

As [[Data_Management#Home_Disk_Space | discussed]] [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf elsewhere], <tt>/home</tt> is mounted read-only on the compute nodes; you can only write to <tt>/home</tt> from the login nodes and devel nodes. (The [[GPC_Quickstart#128Glargemem | largemem nodes]] on GPC, in this respect, are more like devel nodes than compute nodes). In general, to run jobs you can read from <tt>/home</tt> but you'll have to write to <tt>/scratch</tt> (or, if you were allocated space through the LRAC/NRAC process, on <tt>/project</tt>). More information on SciNet filesytems can be found on our [[Data_Management | Data Management]] page.

===Error Submitting My Job: qsub: Bad UID for job execution MSG=ruserok failed ===

I write up a submission script as in the examples, but when I attempt to submit the job, I get the above error. What's wrong?

'''Answer:'''

This error will occur if you try to submit a job from the login nodes. The login nodes are the gateway to all of SciNet's systems (GPC, TCS, P7, ARC), which have different hardware and queueing systems. To submit a job, you must log into a development node for the particular cluster you are submitting to and submit from there.

===OpenMP on the TCS===

How do I run an OpenMP job on the TCS?

'''Answer:'''

Please look at the [[TCS_Quickstart#Submission_Script_for_an_OpenMP_Job | TCS Quickstart ]] page.

===Can I can use hybrid codes consisting of MPI and openMP on the GPC?===

'''Answer:'''

Yes. Please look at the [[GPC_Quickstart#Hybrid_MPI.2FOpenMP_jobs | GPC Quickstart ]] page.

===How do I run serial jobs on GPC?===

'''Answer''':

So it should be said first that SciNet is a parallel computing resource,
and our priority will always be parallel jobs. Having said that, if
you can make efficient use of the resources using serial jobs and get
good science done, that's good too, and we're happy to help you.

The GPC nodes each have 8 processing cores, and making efficient use of these
nodes means using all eight cores. As a result, we'd like to have the
users take up whole nodes (eg, run multiples of 8 jobs) at a time.

It depends on the nature of your job what the best strategy is. Several approaches are presented on the [[User_Serial|serial run wiki page]].

===Why can't I request only a single cpu for my job on GPC?===

'''Answer''':

On GPC, computers are allocated by the node - that is, in chunks of 8 processors. If you want to run a job that requires only one processor, you need to bundle the jobs into groups of 8, so as to not be wasting the other 7 for 48 hours. See [[User_Serial|serial run wiki page]].

===How do I run serial jobs on TCS?===

'''Answer''': You don't.

===But in the queue I found a user who is running jobs on GPC, each of which is using only one processor, so why can't I?===

'''Answer''':

The pradat* and atlaspt* jobs, amongst others, are jobs of the ATLAS high energy physics project. That they are reported as single cpu jobs is an artifact of the moab scheduler. They are in fact being automatically bundled into 8-job bundles but have to run individually to be compatible with their international grid-based systems.

===How do I use the ramdisk on GPC?===

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) ${SCRATCH}. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

''More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].''

===How can I automatically resubmit a job?===

Commonly you may have a job that you know will take longer to run than what is
permissible in the queue. As long as your program contains [[Checkpoints|checkpoint]] or
restart capability, you can have one job automatically submit the next. In
the following example it is assumed that the program finishes before
the 48 hour limit and then resubmits itself by logging into one
of the development nodes.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example submission script for auto resubmission
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code

# RESUBMIT 10 TIMES HERE
num=$NUM
if [ $num -lt 10 ]; then
num=$(($num+1))
ssh gpc01 "cd $PBS_O_WORKDIR; qsub ./script_name.sh -v NUM=$num";
fi
</source>

<pre>
qsub script_name.sh -v
</pre>

You can alternatively use [[ Moab#Job_Dependencies | Job dependencies ]] through the queuing system which will not start one job until another job has completed.

If your job can't be made to automatically stop before the 48 hour queue window, but it does write out checkpoints, you can use the timeout command to stop the program while you still have time to resubmit; for instance

<source lang="bash">
timeout 2850m ./run_my_code argument1 argument2
</source>

will run the program for 47.5 hours (2850 minutes), and then send it SIGTERM to exit the program.

===How can I pass in arguments to my submission script?===

If you wish to make your scripts more generic you can use qsub's ability
to pass in environment variables to pass in arguments to your script.
The following example shows a case where an input and an output
file are passed in on the qsub line. Multiple variables can be
passed in using the qsub "-v" option and comma delimited.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example of passing in arguments
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code -f $INFILE -o $OUTFILE
</source>

<pre>
qsub script_name.sh -v INFILE=input.txt,OUTFILE=outfile.txt
</pre>

=== How can I run a job longer than 48 hours? ===

'''Answer:'''

The SciNet queues have a queue limit of 48 hours. This is pretty typical for systems of its size in Canada and elsewhere, and larger systems commonly have shorter limits. The limits are there to ensure that every user gets a fair share of the system (so that no one user ties up lots of nodes for a long time), and for safety (so that if one memory board in one node fails in the middle of a very long job, you haven't lost a months' worth of work).

Since many of us have simulations that require more than that much time, most widely-used scientific applications have "checkpoint-restart" functionality, where every so often the complete state of the calculation is stored as a checkpoint file, and one can restart a simulation from one of these. In fact, these restart files tend to be quite useful for a number of purposes.

If your job will take longer, you will have to submit your job in multiple parts, restarting from a checkpoint each time. In this way, one can run a simulation much longer than the queue limit. In fact, one can even write job scripts which automatically re-submit themselves until a run is completed, using [[FAQ#How_can_I_automatically_resubmit_a_job.3F | automatic resubmission. ]]

=== Why did showstart say it would take 3 hours for my job to start before, and now it says my job will start in 10 hours? ===

'''Answer:'''

Please look at the [[FAQ#How_do_priorities_work.2Fwhy_did_that_job_jump_ahead_of_mine_in_the_queue.3F | How do priorities work/why did that job jump ahead of mine in the queue? ]] page.

===How do priorities work/why did that job jump ahead of mine in the queue?===

'''Answer:'''

The [[Moab | queueing system]] used on SciNet machines is a [http://en.wikipedia.org/wiki/Priority_queue Priority Queue]. Jobs enter the queue at the back of the queue, and slowly make their way to the front as those ahead of them are run; but a job that enters the queue with a higher priority can `cut in line'.

The main factor which determines priority is whether or not the user (or their PI) has an [http://wiki.scinethpc.ca/wiki/index.php/Application_Process LRAC or NRAC allocation]. These are competitively allocated grants of computer time; there is a call for proposals towards the end of every calendar year. Users with an allocation have high priorities in an attempt to make sure that they can use the amount of computer time the committees granted them. Their priority decreases as they approach their allotted usage over the current window of time; by the time that they have exhausted that allotted usage, their priority is the same as users with no allocation (unallocated, or `default' users). Unallocated users have a fixed, low, priority.

This priority system is called `fairshare'; the scheduler attempts to make sure everyone has their fair share of the machines, where the share that's fair has been determined by the allocation committee. The fairshare window is a rolling window of two weeks; that is, any time you have a job in the queue, the fairshare calculation of its priority is given by how much of your allocation of the machine has been used in the last 14 days.

A particular allocation might have some fraction of GPC - say 4% of the machine (if the PI had been allocated 10 million CPU hours on GPC). The allocations have labels; (called `Resource Allocation Proposal Identifiers', or RAPIs) they look something like

abc-123-ab

where abc-123 is the PIs CCRI, and the suffix specifies which of the allocations granted to the PI is to be used. These can be specified on a job-by-job basis. On GPC, one adds the line
#PBS -A RAPI
to your script; on TCS, one uses
# @ account_no = RAPI
If the allocation to charge isn't specified, a default is used; each user has such a default, which can be changed at the same portal where one changes one's password,

https://portal.scinet.utoronto.ca/

A jobs priority is determined primarily by the fairshare priority of the allocation it is being charged to; the previous 14 days worth of use under that allocation is calculated and compared to the allocated fraction (here, 5%) of the machine over that window (here, 14 days). The fairshare priority is a decreasing function of the allocation left; if there is no allocation left (eg, jobs running under that allocation have already used 379,038 CPU hours in the past 14 days), the priority is the same as that of a user with no granted allocation. (This last part has been the topic of some debate; as the machine gets more utilized, it will probably be the case that we allow RAC users who have greatly overused their quota to have their priorities to drop below that of unallocated users, to give the unallocated users some chance to run on our increasingly crowded system; this would have no undue effect on our allocated users as they still would be able to use the amount of resources they had been allocated by the committees.) Note that all jobs charging the same allocation get the same fairshare priority.

There are other factors that go into calculating priority, but fairshare is the most significant. Other factors include
* amount of time waiting in queue (measured in units of the requested runtime). A job that requests 1 hour in the queue and has been waiting 2 days will get a bump in its priority larger than a job that requests 2 days and has been waiting the same time.
* User adjustment of priorities ( See below ).

The major effect of these subdominant terms is to shuffle the order of jobs running under the same allocation.

===How do we manage job priorities within our research group?===

'''Answer:'''

Obviously, managing shared resources within a large group - whether it
is conference funding or CPU time - takes some doing.

It's important to note that the fairshare periods are intentionally kept
quite short - just two weeks long. So, for example, let us say that in your resource
allocation you have about 10% of the machine. Then for someone to use
up the whole two week amount of time in 2 days, they'd have to use 70%
of the machine in those two days - which is unlikely to happen by
accident. If that does happen,
those using the same allocation as the person who used 70% of the
machine over the two days will suffer by having much lower priority for
their jobs, but only for the next 12 days - and even then, if there are
idle cpus they'll still be able to compute.

There will be online tools for seeing how the allocation is being used,
and those people who are in charge in your group will be able to use
that information to manage the users, telling them to dial it down or
up. We know that managing a large research group is hard, and we want
to make sure we provide you the information you need to do your job
effectively.

One way for users within a group to manage their priorities within the group
is with [[Moab#Adjusting_Job_Priority | user-adjusted priorities]]; this is
described in more detail on the [[Moab | Scheduling System]] page.

=== How do I charge jobs to my NRAC/LRAC allocation? ===

'''Answer:'''

Please see the [[Moab#Accounting|accounting section of Moab page]].

=== How does one check the amount of used CPU-hours in a project, and how does one get statistics for each user in the project? ===

'''Answer:'''

This information is available on the scinet portal,https://portal.scinet.utoronto.ca, See also [[SciNet Usage Reports]].

=== How does the Infiniband Upgrade affect my 2012 NRAC allocation ?===

The NRAC allocations for the current (2012) year that were based on ethernet and infiniband will carry over, however the allocation will be on the full GPC, not just the subsection. So if you were allocated 500 hours on Infiniband your fairshare allocation will still be 500 hours, just 500 out or 30,000, instead of 500 out of 7,000. If you received two allocations, one on gigE and one on IB, they will simply be combined. This should benefit all users as the desegregation of the GPC provides a greater pool of nodes increasing the probability of your job to run.

==Monitoring jobs in the queue==

===Why hasn't my job started?===

'''Answer:'''

Use the moab command

<pre>
checkjob -v jobid
</pre>

and the last couple of lines should explain why a job hasn't started.

Please see [[Moab| Job Scheduling System (Moab) ]] for more detailed information

===How do I figure out when my job will run?===

'''Answer:'''

Please see [[Moab#Available_Resources| Job Scheduling System (Moab) ]]



===I submit my GPC job, and I get an email saying it was rejected===
'''Answer:'''

This happens because the job you've submitted breaks one of the rules of the queues and is rejected. An email
is sent with the JOBID, JOBNAME, and the reason it was rejected. The following is an example where a job
requests more than 48 hours and was rejected.

<pre>
PBS Job Id: 3462493.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'wclimit too high for class 'batch_ib' (345600 > 172800)'
</pre>

Jobs on the TCS or GPC may only run for 48 hours at a time; this restriction greatly increases responsiveness of the queue and queue throughput for all our users. If your computation requires longer than that, as many do, you will have to [[ Checkpoints | checkpoint ]] your job and restart it after each 48-hour queue window. You can manually re-submit jobs, or if you can have your job cleanly exit before the 48 hour window, there are ways to [[ FAQ#How_can_I_automatically_resubmit_a_job.3F | automatically resubmit jobs ]].

Other rejections return a more cryptic error saying "job violates class configuration" such as follows:
<pre>
PBS Job Id: 3462409.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'user required by class 'batch''
</pre>

The most common problems that result in this error are:

* '''Incorrect number of processors per node''': Jobs on the GPC are scheduled per-node not per-core and since each node has 8 processor cores (ppn=8) the smallest job allowed is one node with 8 cores (nodes=1:ppn=8). For serial jobs users must bundle or batch them together in groups of 8. See [[ FAQ#How_do_I_run_serial_jobs_on_GPC.3F | How do I run serial jobs on GPC? ]]
* '''No number of nodes specified''': Jobs submitted to the main queue must request a specific number of nodes, either in the submission script (with a line like <tt>#PBS -l nodes=2:ppn=8</tt>) or on the command line (eg, <tt>qsub -l nodes=2:ppn=8,walltime=5:00:00 script.pbs</tt>). Note that for the debug queue, you can get away without specifying a number of nodes and a default of one will be assigned; for both technical and policy reasons, we do not enforce such a default for the main ("batch") queue.
* '''There is a 15 minute walltime minimum''' on all queues except debug and if you set your walltime less than this, it will be rejected.

===How can I monitor my running jobs on TCS?===

How can I monitor the load of TCS jobs?

'''Answer:'''

You can get more information with the command
/xcat/tools/tcs-scripts/LL/jobState.sh
which I alias as:
alias llq1='/xcat/tools/tcs-scripts/LL/jobState.sh'
If you run "llq1 -n" you will see a listing of jobs together with a lot of information, including the load.

==Errors in running jobs==

===On GPC, `Job cannot be executed'===

I get error messages like this trying to run on GPC:

<pre>
PBS Job Id: 30414.gpc-sched
Job Name: namd
Exec host: gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0
Aborted by PBS Server
Job cannot be executed
See Administrator for help

PBS Job Id: 30414.gpc-sched
Job Name: namd
Exec host: gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0
An error has occurred processing your job, see below.
request to copy stageout files failed on node 'gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0' for job 30414.gpc-sched

Unable to copy file 30414.gpc-sched.OU to USER@gpc-f101n084.scinet.local:/scratch/G/GROUP/USER/projects/sim-performance-test/runtime/l/namd/8/namd.o30414
*** error from copy
30414.gpc-sched.OU: No such file or directory
*** end error output
</pre>

Try doing the following:
<pre>
mkdir ${SCRATCH}/.pbs_spool
ln -s ${SCRATCH}/.pbs_spool ~/.pbs_spool
</pre>

This is how all new accounts are setup on SciNet.

<tt>/home</tt> on GPC for compute jobs is mounted as a read-only file system.
PBS by default tries to spool its output files to <tt>${HOME}/.pbs_spool</tt>
which fails as it tries to write to a read-only file
system. New accounts at SciNet get around this by having ${HOME}/.pbs_spool
point to somewhere appropriate on <tt>/scratch</tt>, but if you've deleted that link
or directory, or had an old account, you will see errors like the above.

'''On Feb 24, the input/output mechanism has been reconfigured to use a local ramdisk as the temporary location, which means that .pbs_spool is no longer needed and this error should not occur anymore.'''

=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in
/var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

=== My GPC job died, telling me `Copy Stageout Files Failed' ===

'''Answer:'''

When a job runs on GPC, the script's standard output and error are redirected to
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:
<pre>
An error has occurred processing your job, see below.
request to copy stageout files failed on node
</pre>
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process and those files just never got copied.

Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.



===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or
network configurations may exhibit this problem at higher or lower MPI
task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

<pre>
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288
</pre>

With Intel MPI, you should be able to do

<pre>
module load intelmpi/4.0.3.008
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode
</pre>

to the same end.

For more information see [[GPC MPI Versions]].

===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===

'''Answer:'''

To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.

==Data on SciNet disks==

===How do I find out my disk usage?===

'''Answer:'''

The standard unix/linux utilities for finding the amount of disk space used by a directory are very slow, and notoriously inefficient on the GPFS filesystems that we run on the SciNet systems. There are utilities that very quickly report your disk usage:

The <tt>'''/scinet/gpc/bin/diskUsage'''</tt> command, available on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time.
This information is only updated hourly!

More information about these filesystems is available at the [[Data_Management | Data_Management]].

===How do I transfer data to/from SciNet?===

'''Answer:'''

All incoming connections to SciNet go through relatively low-speed connections to the <tt>login.scinet</tt> gateways, so using scp to copy files the same way you ssh in is not an effective way to move lots of data. Better tools are described in our page on [[Data_Management#Data_Transfer | Data Transfer]].

===My group works with data files of size 1-2 GB. Is this too large to transfer by scp to login.scinet.utoronto.ca ?===

'''Answer:'''

Generally, occasion transfers of data less than 10GB is perfectly acceptible to so through the login nodes. See [[Data_Management#Data_Transfer | Data Transfer]].

===How can I check if I have files in /scratch that are scheduled for automatic deletion?===

'''Answer:'''

Please see [[Storage_Quickstart#Scratch_Disk_Purging_Policy | Storage At SciNet]]

===How to allow my supervisor to manage files for me using ACL-based commands?===

'''Answer:'''

Please see [[Data_Management#File.2FOwnership_Management_.28ACL.29 | File/Ownership Management]]

===Can we buy extra storage space on SciNet?===

'''Answer:'''
Yes, please see [[Data_Management#Buying_storage_space_on_GPFS_or_HPSS | Buying storage space on GPFS or HPSS ]] for more details.

==Keep 'em Coming!==

===Next question, please===

Send your question to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]; we'll answer it asap!

FAQ

2013-07-16T17:04:42Z

Ljdursi: /* When will the 2011 NRAC disk space allocation be ready? */ - time to delete this.

__TOC__

==The Basics==
===Whom do I contact for support?===

Whom do I contact if I have problems or questions about how to use the SciNet systems?

'''Answer:'''

E-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

In your email, please include the following information:

* your username on SciNet
* the cluster that your question pertains to (GPC or TCS; SciNet is not a cluster!),
* any relevant error messages
* the commands you typed before the errors occured
* the path to your code (if applicable)
* the location of the job scripts (if applicable)
* the directory from which it was submitted (if applicable)
* a description of what it is supposed to do (if applicable)
* if your problem is about connecting to SciNet, the type of computer you are connecting from.

Note that your password should never, never, never be to sent to us, even if your question is about your account.

Try to avoid sending email only to specific individuals at SciNet. Your chances of a quick reply increase significantly if you email our team!

===What does ''code scaling'' mean?===

'''Answer:'''

Please see [[Introduction_To_Performance#Parallel_Speedup|A Performance Primer]]

===What do you mean by ''throughput''?===

'''Answer:'''

Please see [[Introduction_To_Performance#Throughput|A Performance Primer]].

Here is a simple example:

Suppose you need to do 10 computations. Say each of these runs for
1 day on 8 cores, but they take "only" 18 hours on 16 cores. What is the
fastest way to get all 10 computations done - as 8-core jobs or as
16-core jobs? Let us assume you have 2 nodes at your disposal.
The answer, after some simple arithmetic, is that running your 10
jobs as 8-core jobs will take 5 days, whereas if you ran them
as 16-core jobs it would take 7.5 days. Take your own conclusions...

===I changed my .bashrc/.bash_profile and now nothing works===

The default startup scripts provided by SciNet, and guidelines for them, can be found [[Important_.bashrc_guidelines|here]]. Certain things - like sourcing <tt>/etc/profile</tt>
and <tt>/etc/bashrc</tt> are ''required'' for various SciNet routines to work!

If the situation is so bad that you cannot even log in, please send email [mailto:support@scinet.utoronto.ca support].

===Could I have my login shell changed to (t)csh?===

The login shell used on our systems is bash. While the tcsh is available on the GPC and the TCS, we do not support it as the default login shell at present. So "chsh" will not work, but you can always run tcsh interactively. Also, csh scripts will be executed correctly provided that they have the correct "shebang" <tt>#!/bin/tcsh</tt> at the top.

===How can I run Matlab / IDL / Gaussian / my favourite commercial software at SciNet?===

'''Answer:'''

Because SciNet serves such a disparate group of user communities, there is just no way we can buy licenses for everyone's commercial package. The only commercial software we have purchased is that which in principle can benefit everyone -- fast compilers and math libraries (Intel's on GPC, and IBM's on TCS).

If your research group requires a commercial package that you already have or are willing to buy licenses for, contact us at [mailto:support@scinet.utoronto.ca support@scinet] and we can work together to find out if it is feasible to implement the packages licensing arrangement on the SciNet clusters, and if so, what is the the best way to do it.

Note that it is important that you contact us before installing commercially licensed software on SciNet machines, even if you have a way to do it in your own directory without requiring sysadmin intervention. It puts us in a very awkward position if someone is found to be running unlicensed or invalidly licensed software on our systems, so we need to be aware of what is being installed where.

===Do you have a recommended ssh program that will allow scinet access from Windows machines?===

'''Answer:'''

The [[Ssh#SSH_for_Windows_Users | SSH for Windows users]] programs we recommend are:

* [http://mobaxterm.mobatek.net/en/ MobaXterm] is a tabbed ssh client with some Cygwin tools, including ssh and X, all wrapped up into one executable.
* [http://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY] - this is a terminal for windows that connects via ssh. It is a quick install and will get you up and running quickly. To set up your passphrase protected ssh key with putty, see [http://the.earth.li/~sgtatham/putty/0.61/htmldoc/Chapter8.html#pubkey here].
* [http://www.cygwin.com/ CygWin] - this is a whole linux-like environment for windows, which also includes an X window server so that you can display remote windows on your desktop. Make sure you include the openssh and X window system in the installation for full functionality. This is recommended if you will be doing a lot of work on Linux machines, as it makes a very similar environment available on your computer. To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.
 To set up your ssh keys, following the Linux instruction on the [[Ssh keys]] page.

===My ssh key does not work! WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! ===

'''Answer:'''

[[Ssh_keys#Testing_Your_Key | Testing Your Key]]

* If this doesn't work, you should be able to login using your password, and investigate the problem. For example, if during a login session you get an message similar to the one below, just follow the instruction and delete the offending key on line 3 (you can use vi to jump to that line with ESC plus : plus 3). That only means that you may have logged in from your home computer to SciNet in the past, and that key is obsolete.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@**@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle
attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
53:f9:60:71:a8:0b:5d:74:83:52:**fe:ea:1a:9e:cc:d3.
Please contact your system administrator.
Add correct host key in /home/<user>/.ssh/known_hosts to get rid of
this message.
Offending key in /home/<user>/.ssh/known_hosts:3
RSA host key for login.scinet.utoronto.ca
<http://login.scinet.utoronto.ca <http://login.scinet.utoronto.ca>> has
changed and you have requested
</pre>

* If you get the message below you may need to logout of your gnome session and log back in since the ssh-agent needs to be
restarted with the new passphrase ssh key.
<pre>
$ ssh USERNAME@login.scinet.utoronto.ca

Agent admitted failure to sign using the key.
</pre>

===Can't forward X: "Warning: No xauth data; using fake authentication data", or "X11 connection rejected because of wrong authentication."===

I used to be able to forward X11 windows from SciNet to my home machine, but now I'm getting these messages; what's wrong?

'''Answer:'''

This very likely means that ssh/xauth can't update your ${HOME}/.Xauthority file.

The simplest pssible reason for this is that you've filled your 10GB /home quota and so can't write anything to your home directory. Use

<pre>
$ module load extras
$ diskUsage
</pre>

to check to see how close you are to your disk usage on ${HOME}.

Alternately, this could mean your .Xauthority file has become broken/corrupted/confused some how, in which case you can delete that file, and when you next log in you'll get a similar warning message involving creating .Xauthority, but things should work.

===How come I can not login to TCS?===

'''Answer:'''

A SciNet account doesn't automatically entitle you to TCS access. At a minimum, TCS jobs need to run on at least 32 cores (64 preferred because of Simultaneous Multi Threading - [[TCS_Quickstart#Node_configuration|SMT]] - on these nodes) and need the large memory (4GB/core) and bandwidth on the system. Essentially you need to be able to explain why the work can't be done on the GPC.

===How can I reset the password for my Compute Canada account?===

'''Answer:'''

You can reset your password for your Compute Canada account here:

https://ccdb.computecanada.org/security/forgot

===How can I change or reset the password for my SciNet account?===

'''Answer:'''

To reset your password at SciNet please e-mail [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]

If you know your old password and want to change it, that can be done here:

https://portal.scinet.utoronto.ca/

===Why am I getting the error "Permission denied (publickey,gssapi-with-mic,password)"?===

This error can pop up in a variety of situations: when trying to log in, or when after a job has finished, when the error and output files fail to be copied (there are other possible reasons for this failure as well -- see [[FAQ#My_GPC_job_died.2C_telling_me_.60Copy_Stageout_Files_Failed.27|My GPC job died, telling me:Copy Stageout Files Failed]]).
In most cases, the "Permission denioed" error is caused by incorrect permission of the (hidden) .ssh directory. Ssh is used for logging in as well as for the copying of the standard error and output files after a job.

For security reasons,
the directory .ssh should only be writable and readable to you, but yours
has read permission for everybody, and thus it fails. You can change
this by
<pre>
chmod 700 ~/.ssh
</pre>
And to be sure, also do
<pre>
chmod 600 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub ~/authorized_keys
</pre>

===ERROR:102: Tcl command execution failed? when loading modules ===
Modules sometimes require other modules to be loaded first.
Module will let you know if you didn’t.
For example:
<pre>
$ module purge
$ module load python
python/2.6.2(11):ERROR:151: Module ’python/2.6.2’ depends on one of the module(s) ’gcc/4.4.0’
python/2.6.2(11):ERROR:102: Tcl command execution failed: prereq gcc/4.4.0
$ gpc-f103n084-$ module load gcc python
$
</pre>

==Compiling your Code==

===How can I get g77 to work?===

The fortran 77 compilers on the GPC are ifort and gfortran. We have dropped support for g77. This has been a conscious decision. g77 (and the associated library libg2c) were completely replaced six years ago (Apr 2005) by the gcc 4.x branch, and haven't undergone any updates at all, even bug fixes, for over five years.
If we would install g77 and libg2c, we would have to deal with the inevitable confusion caused when users accidentally link against the old, broken, wrong versions of the gcc libraries instead of the correct current versions.

If your code for some reason specifically requires five-plus-year-old libraries, availability, compatibility, and unfixed-known-bug problems are only going to get worse for you over time, and this might be as good an opportunity as any to address those issues.

''A note on porting to gfortran or ifort:''

While gfortran and ifort are rather compatible with g77, one
important difference is that by default, gfortran does not preserve
local variables between function calls, while g77 does. Preserved
local variables are for instance often used in implementations of quasi-random number
generators. Proper fortran requires to declare such variables as SAVE
but not all old code does this.
Luckily, you can change gfortran's default behavior with the flag
<tt>-fno-automatic</tt>. For ifort, the corresponding flag is <tt>-noautomatic</tt>.

===Where is libg2c.so?===

libg2c.so is part of the g77 compiler, for which we dropped support. See [[#How can I get g77 to work on the GPC?]] for our reasons.

===Autoparallelization does not work!===

I compiled my code with the <tt>-qsmp=omp,auto</tt> option, and then I specified that it should be run with 64 threads - with
export OMP_NUM_THREADS=64

However, when I check the load using <tt>llq1 -n</tt>, it shows a load on the node of 1.37. Why?

'''Answer:'''

Using the autoparallelization will only get you so far. In fact, it usually does not do too much. What is helpful is to run the compiler with the <tt>-qreport</tt> option, and then read the output listing carefully to see where the compiler thought it could parallelize, where it could not, and the reasons for this. Then you can go back to your code and carefully try to address each of the issues brought up by the compiler.
We ''emphasize'' that this is just a rough first guide, and that the compilers are still not magical! For more sophisticated approaches to parallelizing your code, email us at [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] to set up an appointment with one
of our technical analysts.

===How do I link against the Intel Math Kernel Library?===

If you need to link in the Intel Math Kernel Library (MKL) libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code.

'''''Note that this give the link line for the command line. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''''

'''''Note too that, unless the integer arguments you will be passing to the MKL libraries are actually 64-bit integers, rather than the normal int or INTEGER types, you want to specify 32-bit integers (lp64) .'''''

===Can the compilers on the login nodes be disabled to prevent accidentally using them?===

'''Answer:'''

You can accomplish this by modifying your .bashrc to not load the compiler modules. See [[Important .bashrc guidelines]].

==="relocation truncated to fit: R_X86_64_PC32": Huh?===

What does this mean, and why can't I compile this code?

'''Answer:'''

Welcome to the joys of the x86 architecture! You're probably having trouble building arrays larger than 2GB, individually or together. Generally, you have to try to use the medium or large x86 `memory model'. For the intel compilers, this is specified with the compile options

-mcmodel=medium -shared-intel

==="feupdateenv is not implemented and will always fail"===

How do I get rid of this and what does it mean?

'''Answer:'''
First note that, as ominous as it sounds, this is really just a warning, and has to do with the intel math library. You can ignore it (unless you really are trying to manually change the exception handlers for floating point exceptions such as divide by zero), or take the safe road and get rid off it by linking with the intel math functions library:<pre>-limf</pre>See also [[#How do I link against the Intel Math Kernel Library?]]

===Cannot find rdmacm library when compiling on GPC===

I get the following error building my code on GPC: "<tt>ld: cannot find -lrdmacm</tt>". Where can I find this library?

'''Answer:'''

This library is part of the MPI libraries; if your compiler is having problems picking it up, it probably means you are mistakenly trying to compile on the login nodes (scinet01..scinet04). The login nodes aren't part of the GPC; they are for logging into the data centre only. From there you must go to the GPC or TCS development nodes to do any real work.

=== Why do I get this error when I try to compile: "icpc: error #10001: could not find directory in which /usr/bin/g++41 resides" ?===

You are trying to compile on the login nodes. As described in the wiki ( https://support.scinet.utoronto.ca/wiki/index.php/GPC_Quickstart#Login ), or in the users guide you would have received with your account, Scinet supports two main clusters, with very different architectures. Compilation must be done on the development nodes of the appropriate cluster (in this case, gpc01-04). Thus, log into gpc01, gpc02, gpc03, or gpc04, and compile from there.

==Testing your Code==

=== Can I run a something for a short time on the development nodes? ===

I am in the process of playing around with the mpi calls in my code to get it to work. I do a lot of tests and each of them takes a couple of seconds only. Can I do this on the development nodes?

'''Answer:'''

Yes, as long as it's very brief (a few minutes). People use the development nodes
for their work, and you don't want to bog it down for people, and testing a real
code can chew up a lot more resources than compiling, etc. The procedures differ
depending on what machine you're using.

==== TCS ====

On the TCS you can run small MPI jobs on the tcs02 node, which is meant for
development use. But even for this test run on one node, you'll need a host file --
a list of hosts (in this case, all tcs-f11n06, which is the `real' name of tcs02)
that the job will run on. Create a file called `hostfile' containing the following:

tcs-f11n06
tcs-f11n06
tcs-f11n06
tcs-f11n06

for a 4-task run. When you invoke "poe" or "mpirun", there are runtime
arguments that you specify pointing to this file. You can also specify it
in an environment variable MP_HOSTFILE, so, if your file is in your /scratch directory, say
${SCRATCH}/hostfile, then you would do

<pre>
export MP_HOSTFILE=${SCRATCH}/hostfile
</pre>

in your shell. You will also need to create a <tt>.rhosts</tt> file in your
home director, again listing <tt>tcs-f11n06</tt> so that <tt>poe</tt>
can start jobs. After that you can simply run your program. You can use
mpiexec:

<pre>
mpiexec -n 4 my_test_program
</pre>

adding <tt> -hostfile /path/to/my/hostfile</tt> if you did not set the environment
variable above. Alternatively, you can run it with the poe command (do a "man poe" for details), or even by
just directly running it. In this case the number of MPI processes will by default
be the number of entries in your hostfile.

==== GPC ====

On the GPC one can run short test jobs on the GPC [[GPC_Quickstart#Compile.2FDevel_Nodes | development nodes ]]<tt>gpc01</tt>..<tt>gpc04</tt>;
if they are single-node jobs (which they should be) they don't need a hostfile. Even better, though, is to request an [[ Moab#Interactive | interactive ]] job and run the tests either in regular batch queue or using a short high availability [[ Moab#debug | debug ]] queue that is reserved for this purpose.

=== How do I run a longer (but still shorter than an hour) test job quickly ? ===

'''Answer'''

On the GPC there is a high turnover short queue called [[ Moab#debug | debug ]] that is designed for
this purpose. You can use it by adding
<pre>
#PBS -q debug
</pre>
to your submission script.

==Running your jobs==

===My job can't write to /home===

My code works fine when I test on the development nodes, but when I submit a job, or even run interactively in the development queue on GPC, it fails. What's wrong?

'''Answer:'''

As [[Data_Management#Home_Disk_Space | discussed]] [https://support.scinet.utoronto.ca/wiki/images/5/54/SciNet_Tutorial.pdf elsewhere], <tt>/home</tt> is mounted read-only on the compute nodes; you can only write to <tt>/home</tt> from the login nodes and devel nodes. (The [[GPC_Quickstart#128Glargemem | largemem nodes]] on GPC, in this respect, are more like devel nodes than compute nodes). In general, to run jobs you can read from <tt>/home</tt> but you'll have to write to <tt>/scratch</tt> (or, if you were allocated space through the LRAC/NRAC process, on <tt>/project</tt>). More information on SciNet filesytems can be found on our [[Data_Management | Data Management]] page.

===Error Submitting My Job: qsub: Bad UID for job execution MSG=ruserok failed ===

I write up a submission script as in the examples, but when I attempt to submit the job, I get the above error. What's wrong?

'''Answer:'''

This error will occur if you try to submit a job from the login nodes. The login nodes are the gateway to all of SciNet's systems (GPC, TCS, P7, ARC), which have different hardware and queueing systems. To submit a job, you must log into a development node for the particular cluster you are submitting to and submit from there.

===OpenMP on the TCS===

How do I run an OpenMP job on the TCS?

'''Answer:'''

Please look at the [[TCS_Quickstart#Submission_Script_for_an_OpenMP_Job | TCS Quickstart ]] page.

===Can I can use hybrid codes consisting of MPI and openMP on the GPC?===

'''Answer:'''

Yes. Please look at the [[GPC_Quickstart#Hybrid_MPI.2FOpenMP_jobs | GPC Quickstart ]] page.

===How do I run serial jobs on GPC?===

'''Answer''':

So it should be said first that SciNet is a parallel computing resource,
and our priority will always be parallel jobs. Having said that, if
you can make efficient use of the resources using serial jobs and get
good science done, that's good too, and we're happy to help you.

The GPC nodes each have 8 processing cores, and making efficient use of these
nodes means using all eight cores. As a result, we'd like to have the
users take up whole nodes (eg, run multiples of 8 jobs) at a time.

It depends on the nature of your job what the best strategy is. Several approaches are presented on the [[User_Serial|serial run wiki page]].

===Why can't I request only a single cpu for my job on GPC?===

'''Answer''':

On GPC, computers are allocated by the node - that is, in chunks of 8 processors. If you want to run a job that requires only one processor, you need to bundle the jobs into groups of 8, so as to not be wasting the other 7 for 48 hours. See [[User_Serial|serial run wiki page]].

===How do I run serial jobs on TCS?===

'''Answer''': You don't.

===But in the queue I found a user who is running jobs on GPC, each of which is using only one processor, so why can't I?===

'''Answer''':

The pradat* and atlaspt* jobs, amongst others, are jobs of the ATLAS high energy physics project. That they are reported as single cpu jobs is an artifact of the moab scheduler. They are in fact being automatically bundled into 8-job bundles but have to run individually to be compatible with their international grid-based systems.

===How do I use the ramdisk on GPC?===

To use the ramdisk, create and read to / write from files in /dev/shm/.. just as one would to (eg) ${SCRATCH}. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

It is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

''More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].''

===How can I automatically resubmit a job?===

Commonly you may have a job that you know will take longer to run than what is
permissible in the queue. As long as your program contains [[Checkpoints|checkpoint]] or
restart capability, you can have one job automatically submit the next. In
the following example it is assumed that the program finishes before
the 48 hour limit and then resubmits itself by logging into one
of the development nodes.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example submission script for auto resubmission
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code

# RESUBMIT 10 TIMES HERE
num=$NUM
if [ $num -lt 10 ]; then
num=$(($num+1))
ssh gpc01 "cd $PBS_O_WORKDIR; qsub ./script_name.sh -v NUM=$num";
fi
</source>

<pre>
qsub script_name.sh -v
</pre>

You can alternatively use [[ Moab#Job_Dependencies | Job dependencies ]] through the queuing system which will not start one job until another job has completed.

If your job can't be made to automatically stop before the 48 hour queue window, but it does write out checkpoints, you can use the timeout command to stop the program while you still have time to resubmit; for instance

<source lang="bash">
timeout 2850m ./run_my_code argument1 argument2
</source>

will run the program for 47.5 hours (2850 minutes), and then send it SIGTERM to exit the program.

===How can I pass in arguments to my submission script?===

If you wish to make your scripts more generic you can use qsub's ability
to pass in environment variables to pass in arguments to your script.
The following example shows a case where an input and an output
file are passed in on the qsub line. Multiple variables can be
passed in using the qsub "-v" option and comma delimited.

<source lang="bash">
#!/bin/bash
# MOAB/Torque example of passing in arguments
# SciNet GPC
#
#PBS -l nodes=1:ppn=8,walltime=48:00:00
#PBS -N my_job

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# YOUR CODE HERE
./run_my_code -f $INFILE -o $OUTFILE
</source>

<pre>
qsub script_name.sh -v INFILE=input.txt,OUTFILE=outfile.txt
</pre>

=== How can I run a job longer than 48 hours? ===

'''Answer:'''

The SciNet queues have a queue limit of 48 hours. This is pretty typical for systems of its size in Canada and elsewhere, and larger systems commonly have shorter limits. The limits are there to ensure that every user gets a fair share of the system (so that no one user ties up lots of nodes for a long time), and for safety (so that if one memory board in one node fails in the middle of a very long job, you haven't lost a months' worth of work).

Since many of us have simulations that require more than that much time, most widely-used scientific applications have "checkpoint-restart" functionality, where every so often the complete state of the calculation is stored as a checkpoint file, and one can restart a simulation from one of these. In fact, these restart files tend to be quite useful for a number of purposes.

If your job will take longer, you will have to submit your job in multiple parts, restarting from a checkpoint each time. In this way, one can run a simulation much longer than the queue limit. In fact, one can even write job scripts which automatically re-submit themselves until a run is completed, using [[FAQ#How_can_I_automatically_resubmit_a_job.3F | automatic resubmission. ]]

=== Why did showstart say it would take 3 hours for my job to start before, and now it says my job will start in 10 hours? ===

'''Answer:'''

Please look at the [[FAQ#How_do_priorities_work.2Fwhy_did_that_job_jump_ahead_of_mine_in_the_queue.3F | How do priorities work/why did that job jump ahead of mine in the queue? ]] page.

===How do priorities work/why did that job jump ahead of mine in the queue?===

'''Answer:'''

The [[Moab | queueing system]] used on SciNet machines is a [http://en.wikipedia.org/wiki/Priority_queue Priority Queue]. Jobs enter the queue at the back of the queue, and slowly make their way to the front as those ahead of them are run; but a job that enters the queue with a higher priority can `cut in line'.

The main factor which determines priority is whether or not the user (or their PI) has an [http://wiki.scinethpc.ca/wiki/index.php/Application_Process LRAC or NRAC allocation]. These are competitively allocated grants of computer time; there is a call for proposals towards the end of every calendar year. Users with an allocation have high priorities in an attempt to make sure that they can use the amount of computer time the committees granted them. Their priority decreases as they approach their allotted usage over the current window of time; by the time that they have exhausted that allotted usage, their priority is the same as users with no allocation (unallocated, or `default' users). Unallocated users have a fixed, low, priority.

This priority system is called `fairshare'; the scheduler attempts to make sure everyone has their fair share of the machines, where the share that's fair has been determined by the allocation committee. The fairshare window is a rolling window of two weeks; that is, any time you have a job in the queue, the fairshare calculation of its priority is given by how much of your allocation of the machine has been used in the last 14 days.

A particular allocation might have some fraction of GPC - say 4% of the machine (if the PI had been allocated 10 million CPU hours on GPC). The allocations have labels; (called `Resource Allocation Proposal Identifiers', or RAPIs) they look something like

abc-123-ab

where abc-123 is the PIs CCRI, and the suffix specifies which of the allocations granted to the PI is to be used. These can be specified on a job-by-job basis. On GPC, one adds the line
#PBS -A RAPI
to your script; on TCS, one uses
# @ account_no = RAPI
If the allocation to charge isn't specified, a default is used; each user has such a default, which can be changed at the same portal where one changes one's password,

https://portal.scinet.utoronto.ca/

A jobs priority is determined primarily by the fairshare priority of the allocation it is being charged to; the previous 14 days worth of use under that allocation is calculated and compared to the allocated fraction (here, 5%) of the machine over that window (here, 14 days). The fairshare priority is a decreasing function of the allocation left; if there is no allocation left (eg, jobs running under that allocation have already used 379,038 CPU hours in the past 14 days), the priority is the same as that of a user with no granted allocation. (This last part has been the topic of some debate; as the machine gets more utilized, it will probably be the case that we allow RAC users who have greatly overused their quota to have their priorities to drop below that of unallocated users, to give the unallocated users some chance to run on our increasingly crowded system; this would have no undue effect on our allocated users as they still would be able to use the amount of resources they had been allocated by the committees.) Note that all jobs charging the same allocation get the same fairshare priority.

There are other factors that go into calculating priority, but fairshare is the most significant. Other factors include
* amount of time waiting in queue (measured in units of the requested runtime). A job that requests 1 hour in the queue and has been waiting 2 days will get a bump in its priority larger than a job that requests 2 days and has been waiting the same time.
* User adjustment of priorities ( See below ).

The major effect of these subdominant terms is to shuffle the order of jobs running under the same allocation.

===How do we manage job priorities within our research group?===

'''Answer:'''

Obviously, managing shared resources within a large group - whether it
is conference funding or CPU time - takes some doing.

It's important to note that the fairshare periods are intentionally kept
quite short - just two weeks long. (These exact numbers subject to
change as the year goes on and we better understand use patterns, but
they're unlikely to change radically). So, for example, let us say that in your resource
allocation you have about 10% of the machine. Then for someone to use
up the whole two week amount of time in 2 days, they'd have to use 70%
of the machine in those two days - which is unlikely to happen by
accident. If that does happen,
those using the same allocation as the person who used 70% of the
machine over the two days will suffer by having much lower priority for
their jobs, but only for the next 12 days - and even then, if there are
idle cpus they'll still be able to compute.

There will be online tools for seeing how the allocation is being used,
and those people who are in charge in your group will be able to use
that information to manage the users, telling them to dial it down or
up. We know that managing a large research group is hard, and we want
to make sure we provide you the information you need to do your job
effectively.

One way for users within a group to manage their priorities within the group
is with [[Moab#Adjusting_Job_Priority | user-adjusted priorities]]; this is
described in more detail on the [[Moab | Scheduling System]] page.

=== How do I charge jobs to my NRAC/LRAC allocation? ===

'''Answer:'''

Please see the [[Moab#Accounting|accounting section of Moab page]].

=== How does one check the amount of used CPU-hours in a project, and how does one get statistics for each user in the project? ===

'''Answer:'''

This information is available on the scinet portal,https://portal.scinet.utoronto.ca, See also [[SciNet Usage Reports]].

=== How does the Infiniband Upgrade affect my 2012 NRAC allocation ?===

The NRAC allocations for the current (2012) year that were based on ethernet and infiniband will carry over, however the allocation will be on the full GPC, not just the subsection. So if you were allocated 500 hours on Infiniband your fairshare allocation will still be 500 hours, just 500 out or 30,000, instead of 500 out of 7,000. If you received two allocations, one on gigE and one on IB, they will simply be combined. This should benefit all users as the desegregation of the GPC provides a greater pool of nodes increasing the probability of your job to run.

==Monitoring jobs in the queue==

===Why hasn't my job started?===

'''Answer:'''

Use the moab command

<pre>
checkjob -v jobid
</pre>

and the last couple of lines should explain why a job hasn't started.

Please see [[Moab| Job Scheduling System (Moab) ]] for more detailed information

===How do I figure out when my job will run?===

'''Answer:'''

Please see [[Moab#Available_Resources| Job Scheduling System (Moab) ]]



===I submit my GPC job, and I get an email saying it was rejected===
'''Answer:'''

This happens because the job you've submitted breaks one of the rules of the queues and is rejected. An email
is sent with the JOBID, JOBNAME, and the reason it was rejected. The following is an example where a job
requests more than 48 hours and was rejected.

<pre>
PBS Job Id: 3462493.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'wclimit too high for class 'batch_ib' (345600 > 172800)'
</pre>

Jobs on the TCS or GPC may only run for 48 hours at a time; this restriction greatly increases responsiveness of the queue and queue throughput for all our users. If your computation requires longer than that, as many do, you will have to [[ Checkpoints | checkpoint ]] your job and restart it after each 48-hour queue window. You can manually re-submit jobs, or if you can have your job cleanly exit before the 48 hour window, there are ways to [[ FAQ#How_can_I_automatically_resubmit_a_job.3F | automatically resubmit jobs ]].

Other rejections return a more cryptic error saying "job violates class configuration" such as follows:
<pre>
PBS Job Id: 3462409.gpc-sched
Job Name: STDIN
job deleted
Job deleted at request of root@gpc-sched
MOAB_INFO: job was rejected - job violates class configuration 'user required by class 'batch''
</pre>

The most common problems that result in this error are:

* '''Incorrect number of processors per node''': Jobs on the GPC are scheduled per-node not per-core and since each node has 8 processor cores (ppn=8) the smallest job allowed is one node with 8 cores (nodes=1:ppn=8). For serial jobs users must bundle or batch them together in groups of 8. See [[ FAQ#How_do_I_run_serial_jobs_on_GPC.3F | How do I run serial jobs on GPC? ]]
* '''No number of nodes specified''': Jobs submitted to the main queue must request a specific number of nodes, either in the submission script (with a line like <tt>#PBS -l nodes=2:ppn=8</tt>) or on the command line (eg, <tt>qsub -l nodes=2:ppn=8,walltime=5:00:00 script.pbs</tt>). Note that for the debug queue, you can get away without specifying a number of nodes and a default of one will be assigned; for both technical and policy reasons, we do not enforce such a default for the main ("batch") queue.
* '''There is a 15 minute walltime minimum''' on all queues except debug and if you set your walltime less than this, it will be rejected.

===How can I monitor my running jobs on TCS?===

How can I monitor the load of TCS jobs?

'''Answer:'''

You can get more information with the command
/xcat/tools/tcs-scripts/LL/jobState.sh
which I alias as:
alias llq1='/xcat/tools/tcs-scripts/LL/jobState.sh'
If you run "llq1 -n" you will see a listing of jobs together with a lot of information, including the load.

==Errors in running jobs==

===On GPC, `Job cannot be executed'===

I get error messages like this trying to run on GPC:

<pre>
PBS Job Id: 30414.gpc-sched
Job Name: namd
Exec host: gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0
Aborted by PBS Server
Job cannot be executed
See Administrator for help

PBS Job Id: 30414.gpc-sched
Job Name: namd
Exec host: gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0
An error has occurred processing your job, see below.
request to copy stageout files failed on node 'gpc-f120n011/7+gpc-f120n011/6+gpc-f120n011/5+gpc-f120n011/4+gpc-f120n011/3+gpc-f120n011/2+gpc-f120n011/1+gpc-f120n011/0' for job 30414.gpc-sched

Unable to copy file 30414.gpc-sched.OU to USER@gpc-f101n084.scinet.local:/scratch/G/GROUP/USER/projects/sim-performance-test/runtime/l/namd/8/namd.o30414
*** error from copy
30414.gpc-sched.OU: No such file or directory
*** end error output
</pre>

Try doing the following:
<pre>
mkdir ${SCRATCH}/.pbs_spool
ln -s ${SCRATCH}/.pbs_spool ~/.pbs_spool
</pre>

This is how all new accounts are setup on SciNet.

<tt>/home</tt> on GPC for compute jobs is mounted as a read-only file system.
PBS by default tries to spool its output files to <tt>${HOME}/.pbs_spool</tt>
which fails as it tries to write to a read-only file
system. New accounts at SciNet get around this by having ${HOME}/.pbs_spool
point to somewhere appropriate on <tt>/scratch</tt>, but if you've deleted that link
or directory, or had an old account, you will see errors like the above.

'''On Feb 24, the input/output mechanism has been reconfigured to use a local ramdisk as the temporary location, which means that .pbs_spool is no longer needed and this error should not occur anymore.'''

=== I couldn't find the .o output file in the .pbs_spool directory as I used to ===

On Feb 24 2011, the temporary location of standard input and output files was moved from the shared file system ${SCRATCH}/.pbs_spool to the
node-local directory /var/spool/torque/spool (which resides in ram). The final location after a job has finished is unchanged,
but to check the output/error of running jobs, users will now have to ssh into the (first) node assigned to the job and look in
/var/spool/torque/spool.

This alleviates access contention to the temporary directory, especially for those users that are running a lot of jobs, and reduces the burden on the file system in general.

Note that it is good practice to redirect output to a file rather than to count on the scheduler to do this for you.

=== My GPC job died, telling me `Copy Stageout Files Failed' ===

'''Answer:'''

When a job runs on GPC, the script's standard output and error are redirected to
<tt>$PBS_JOBID.gpc-sched.OU</tt> and <tt>$PBS_JOBID.gpc-sched.ER</tt> in
/var/spool/torque/spool on the (first) node on which your job is running. At the end of the job, those .OU and .ER files are copied to where the batch script tells them to be copied, by default <tt>$PBS_JOBNAME.o$PBS_JOBID</tt> and<tt>$PBS_JOBNAME.e$PBS_JOBID</tt>. (You can set those filenames to be something clearer with the -e and -o options in your PBS script.)

When you get errors like this:
<pre>
An error has occurred processing your job, see below.
request to copy stageout files failed on node
</pre>
it means that the copying back process has failed in some way. There could be a few reasons for this. The first thing to '''make sure that your .bashrc does not produce any output''', as the output-stageout is performed by bash and further output can cause this to fail.
But it also could have just been a random filesystem error, or it could be that your job failed spectacularly enough to shortcircuit the normal job-termination process and those files just never got copied.

Write to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>] if your input/output files got lost, as we will probably be able to retrieve them for you (please supply at least the jobid, and any other information that may be relevant).

Mind you that it is good practice to redirect output to a file rather than depending on the job scheduler to do this for you.



===IB Memory Errors, eg <tt> reg_mr Cannot allocate memory </tt>===

Infiniband requires more memory than ethernet; it can use RDMA (remote direct memory access) transport for which it sets aside registered memory to transfer data.

In our current network configuration, it requires a _lot_ more memory, particularly as you go to larger process counts; unfortunately, that means you can't get around the "I need more memory" problem the usual way, by running on more nodes. Machines with different memory or
network configurations may exhibit this problem at higher or lower MPI
task counts.

Right now, the best workaround is to reduce the number and size of OpenIB queues, using XRC: with the OpenMPI, add the following options to your mpirun command:

<pre>
-mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32 -mca btl_openib_max_send_size 12288
</pre>

With Intel MPI, you should be able to do

<pre>
module load intelmpi/4.0.3.008
mpirun -genv I_MPI_FABRICS=shm:ofa -genv I_MPI_OFA_USE_XRC=1 -genv I_MPI_OFA_DYNAMIC_QPS=1 -genv I_MPI_DEBUG=5 -np XX ./mycode
</pre>

to the same end.

For more information see [[GPC MPI Versions]].

===My compute job fails, saying <tt>libpng12.so.0: cannot open shared object file</tt> or <tt>libjpeg.so.62: cannot open shared object file</tt>===

'''Answer:'''

To maximize the amount of memory available for compute jobs, the compute nodes have a less complete system image than the development nodes. In particular, since interactive graphics libraries like matplotlib and gnuplot are usually used interactively, the libraries for their use are included in the devel nodes' image but not the compute nodes.

Many of these extra libraries are, however, available in the "extras" module. So adding a "module load extras" to your job submission script - or, for overkill, to your .bashrc - should enable these scripts to run on the compute nodes.

==Data on SciNet disks==

===How do I find out my disk usage?===

'''Answer:'''

The standard unix/linux utilities for finding the amount of disk space used by a directory are very slow, and notoriously inefficient on the GPFS filesystems that we run on the SciNet systems. There are utilities that very quickly report your disk usage:

The <tt>'''/scinet/gpc/bin/diskUsage'''</tt> command, available on the login nodes, datamovers and the GPC devel nodes, provides information in a number of ways on the home, scratch, and project file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time.
This information is only updated hourly!

More information about these filesystems is available at the [[Data_Management | Data_Management]].

===How do I transfer data to/from SciNet?===

'''Answer:'''

All incoming connections to SciNet go through relatively low-speed connections to the <tt>login.scinet</tt> gateways, so using scp to copy files the same way you ssh in is not an effective way to move lots of data. Better tools are described in our page on [[Data_Management#Data_Transfer | Data Transfer]].

===My group works with data files of size 1-2 GB. Is this too large to transfer by scp to login.scinet.utoronto.ca ?===

'''Answer:'''

Generally, occasion transfers of data less than 10GB is perfectly acceptible to so through the login nodes. See [[Data_Management#Data_Transfer | Data Transfer]].

===How can I check if I have files in /scratch that are scheduled for automatic deletion?===

'''Answer:'''

Please see [[Storage_Quickstart#Scratch_Disk_Purging_Policy | Storage At SciNet]]

===How to allow my supervisor to manage files for me using ACL-based commands?===

'''Answer:'''

Please see [[Data_Management#File.2FOwnership_Management_.28ACL.29 | File/Ownership Management]]

===Can we buy extra storage space on SciNet?===

'''Answer:'''
Yes, please see [[Data_Management#Buying_storage_space_on_GPFS_or_HPSS | Buying storage space on GPFS or HPSS ]] for more details.

==Keep 'em Coming!==

===Next question, please===

Send your question to [mailto:support@scinet.utoronto.ca <support@scinet.utoronto.ca>]; we'll answer it asap!

Oldwiki.scinet.utoronto.ca:System Alerts

2013-05-04T15:18:10Z

Ljdursi:

Oldwiki.scinet.utoronto.ca:System Alerts

2013-05-04T15:17:54Z

Ljdursi: /* System Status */

File:Lecture23-2013.pdf

2013-04-09T14:05:43Z

Ljdursi: uploaded a new version of "File:Lecture23-2013.pdf"

File:Lecture23-2013.pdf

2013-04-09T14:04:02Z

Ljdursi:

Scientific Computing Course

2013-04-09T14:03:41Z

Ljdursi: /* Topics */

''This wiki page concerns the 2013 installment of SciNet's Scientific Computing course. Material from the previous installment can be found on [[Scientific Software Development Course]], [[Numerical Tools for Physical Scientists (course)]], and [[High Performance Scientific Computing]]''

__TOC__
=Syllabus=

==About the course==
* Whole-term graduate course
* Prerequisite: basic C, C++ or Fortran experience.
* Will use `C++ light' and Python
* Topics include: Scientific computing and programming skills, Parallel programming, and Hybrid programming.

There are three parts to this course:

# Scientific Software Development: Jan/Feb 2013 ''python, C++, git, make, modular programming, debugging''
# Numerical Tools for Physical Scientists: Feb/Mar 2013 ''modelling, floating point, Monte Carlo, ODE, linear algebra,fft''
# High Performance Scientific Computing: Mar/Apr 2013 ''openmp, mpi and hybrid programming''

Each part consists of eight one-hour lectures, two per week.

These can be taken separately by astrophysics graduate students at the University of Toronto as mini-courses, and by physics graduate students at the University of Toronto as modular courses.

The first two parts count towards the SciNet Certificate in Scientific Computing, while the third part can count towards the SciNet HPC Certificate. For more info about the SciNet Certificates, see http://www.scinethpc.ca/2012/12/scinet-hpc-certificate-program.

==Location and Times==
[http://www.scinethpc.ca/2010/08/contact-us SciNet HeadQuarters] 
256 McCaul Street, Toronto, ON 
Room 229 (Conference Room) 
Tuesdays 11:00 am - 12:00 noon 
Thursdays 11:00 am - 12:00 noon

==Instructors and office hours==

* Ramses van Zon - 256 McCaul Street, Rm 228 - Mondays 3-4pm
* L. Jonathan Dursi - 256 McCaul Street, Rm 216 - Wednesdays 3-4pm

==Grading scheme==

Attendence to lectures.

Four home work sets (i.e., one per week), to be returned by email by 9:00 am the next Thursday.

==Sign up==
Sign up for this graduate course goes through SciNet's course website. The direct link is https://support.scinet.utoronto.ca/courses/?q=node/99. If you do not have a SciNet account but wish to register for this course, please email support@scinet.utoronto.ca . 
Sign up is closed.

=Part 1: Scientific Software Development=

==Prerequisites==

Some programming experience. Some unix prompt experience.

'''Software that you'll need:'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python 2, IPython, Numpy, SciPy and Matplotlib (which you all get if you use the Enthought distribution) installed on your laptop. Links are given at the bottom of this page.

==Dates==

January 15, 17, 22, 24, 29, and 31, 2013 
February 5 and 7, 2013

==Topics (with lecture slides and recordings)==

===''Lecture 1:'' C++ introduction===
:::[[File:Lecture1-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.html]]
:::[[Media:Lecture1-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.mp4 Video recording]

===''Lecture 2:'' More C++, build and version control ===
:::[[File:Lecture2-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.html]]
:::Guest lecturer: Michael Nolta (CITA) for the git portion of the lecture.
:::[[Media:Lecture2-2013.pdf|C++ and Make slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.mp4 C++ and Make video recording]  /   [[Media:Git-Nolta.pdf|Git slides]]  /   [[#HW1|Homework assigment 1]]

===''Lecture 3:'' Python and visualization===
:::[[File:Lecture3-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.html]]
:::[[Media:Lecture3-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.mp4 Video recording]

===''Lecture 4:'' Modular programming, refactoring, testing===
:::[[File:Lecture4-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.html]]
:::[[Media:Lecture4-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.mp4 Video recording]  /   [[#HW2|Homework assigment 2]]
:::[http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc (course project source file)]  /   [http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py (corresponding python movie generator)]

===''Lecture 5:'' Object oriented programming===
:::[[Media:Lecture5-2013.pdf|Slides]]
:::Recordings of this lecture are missing, but you could view the videos of SciNet's [[One-Day Scientific C++ Class]], in particular the parts on classes, polymorphism, and inheritance.

===''Lecture 6:'' ODE, interpolation===
:::[[File:Lecture6-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.html]]
:::[[Media:ScientificComputing2013-Lecture5-ODE.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.mp4 Video recording]  /   [[#HW3|Homework assigment 3]]

===''Lecture 7:'' Development tools: debugging and profiling===
:::[[File:Lecture7-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.html]]
:::[[Media:ScientificComputing2013-Debugging.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.mp4 Video recording]

===''Lecture 8:'' Objects in Python, linking C++ and Python===
:::[[File:Lecture8-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.html]]
:::[[Media:Lecture8-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.mp4 Video recording]

==Homework assignments==

===HW1===

'''''Multi-file C++ program to create a data file'''''

We’ve learned programming in basic C++, use of make and Makefiles to build projects, and local use of git for version control. In this first assignment, you’ll use these to make a multi-file C++ program, built with make, which computes and outputs a data file.

* Start a git repository, and begin writing a C++ program to
:# Get an array size and a standard deviation from user input,
:# Allocate a 2d array (use the code given in lecture 2),
:# Store a 2d Gaussian with a maximum at the centre of the array and given standard deviation (in units of grid points),
:# Output that array to a text file,
:# Free the array, and exit.
* The output text file should contain just the data in text format, with a row of the file corresponding to a row of the array and with whitespace between the numbers.
* The 2d array creation/freeing routines should be in one file (with an associated header file), the gaussian calculation be in another (ditto), and the output routine be in a third, with the main program calling each of these.
* Use a makefile to build your code (add it to the repository).
* You can start with everything in one file, with hardcoded values for sizes and standard deviation and a static array, then refactor things into multiple files, adding the other features.
* As a test, use the ipython executable that came with your Enthought python distribution to read your data and plot it. If your data file is named ‘data.txt’, running the following:
<pre>
$ ipython --pylab
In [1]: data = numpy.genfromtxt('data.txt')
In [2]: contour(data)
</pre>
should give a nice contour plot of a 2-dimensional gaussian.
* Email in your source code, makefile and the "git log" output of all your commits by email by at 9:00 am Thursday Jan 24th, 2013. Please zip or tar these files together as one attachment, with a file name that includes your name and "HW1".

===HW2===
'''''Refactor legacy code to a modular project with unit tests'''''

In class, today, we talked about modular programming and testing, and the project we’ll be working on for the next three weeks. This homework will start advancing on that project by working on the “legacy” code given to us by our supervisor ([http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc]), with a corresponding python plotting script ([http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py]), and whipping it into shape before we start adding new physics.
* Start a git repository for this project, and add the two files.
* Create a Makefile and add it to the repository.
* Since we have no tests, run the program with console output redirected to a file:
:<pre>$ diffuse > original-output.txt</pre>
 ''It turns out the code has a bug that can make the output different when the same code is run again, which obviously would not be good for a baseline test. Replace 'float error;' by 'float error=0.0;' to fix this.''
* Also save the two .npy output files, e.g. to original-data.npy and original-theory.npy. The triplet of files (original-output.txt, original-data.npy and original-theory.npy) serve as a baseline integrated test (add these to repository).
* Then write a 'test' target in your makefile that:
** Runs 'diffuse' with output to a new file.
** Compares the file with the baseline test file, and compare the .npy files.
:: (hint: the unix command diff or cmp can compare files).
* First refactoring: Move the global variables into the main routine.
* ''Chorus: Test your modified code, and commit.''
* Second refactoring: Extract a diffusion operator routine, that gets called from main.
* ''Chorus''
* Create a .cc/.h module for the diffusion operator.
* ''Chorus''
* Add two tests for the diffusion operator: for a constant and for a linear input field (<tt>rho[i][j]=a*i+b*j</tt>). Add these to the test target in the makefile.
* ''Chorus''
* More refactoring: Extract three more .cc/.h modules:
** for output (should not contain hardcoded filenames)
** computation of the theory
** and for the array allocation stuff.
* ''Chorus''
* Describe, but don't implement in the .h and .cc, what would be appropriate unit tests for these three modules.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by 9:00 am on Thursday January 31, 2013.

===HW3===
This week, we learned about object oriented programming, which fits nicely within the modular programming idea. In this homework, we are going to use some of it to restructure our code and get it ready to add the tracer particle, the goal of the course project.

The goal will be to have an instance of a <tt>Diffusion</tt> class,
as well as an instance of <tt>Tracer</tt>, which for now will be a
free particle moving as ('''x'''(t),'''y'''(t)) = ('''x'''(0) +
'''vx''' t, '''y'''(0) + '''vy''' t), without any coupling yet (we
will handle this next week).

To be more specific:
* Clean up your code, using the feedback from your HW2 grading, such that the modules are as independent as possible.
* If you have not done so yet, add comments to the header files of your modules to explain exactly what each function does (without going into implementation details), what its arguments mean and what it returns (unless it's a void function, of course).
* Objectify the <tt>main</tt> routine, by creating a class <tt>Diffusion</tt>.
* Put this class in its own module (declaration in .h, implementation in .cc). For instance, the declaration could be
<pre>
// diffusion.h
#ifndef DIFFUSIONH
#define DIFFUSIONH
#include <fstream>
class Diffusion {
public:
Diffusion(float x1, float x2, float D, int numPoints);
void init(float a0, float sigma0); // set initial field
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
float getRho(int i, int j); // get a value of the field
~Diffusion();
private:
float*** rho;
...
};
#endif
</pre>
(this is not supposed to be prescriptive.)
* In the implementation file you'd have things like
<pre>
// diffusion.cc
#include "diffusion.h"
...
void Diffusion::timeStep(float dt)
{
// code for the timeStep ...
}
...
</pre>
(note the inclusion of the module's header file on the top of the implementation, so the class is declared).
* Let <tt>int main()</tt> have the same functionality as before, but now by defining the parameters of the run, creating an object of this class, setting up file streams, and taking time steps and writing out by using calls to member functions of this object.
* Additionally, write a class <tt>Tracer</tt> which for now implements a free particle in 2d. Something like:
<pre>
class Tracer {
public:
Tracer(float x1, float x2);
void init(float x0, float y0, float vx, float vy);
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
~Tracer();
private:
...
};
</pre>
:The timeStep implementation can in this case use the infamous forward Euler integration scheme, because it happens to be exact here.
:When it comes to output to a npy file, let's view the the data of the tracer particle at one point in time as a 2x2 matrix <tt>[[x,y],[vx,vy]]</tt>, so we can use much of the npy output code that we used for the diffusion field, which was a (numPoints+2)x(numPoints+2) matrix.
* This class too should be its own module (Often, "one class, one module" is a good paradigm, though occasionally you'll have closely related classes).
* Add some code to int main to have the Tracer particle evolve at the same time as the diffusion field (although the two are completely uncoupled).
* Keep using git and make, run the tests that you have regularly to make sure your program still works.

Note that because we've now set up our program in a modular fashion, you can do
different parts of this assignment in any order you want. For instance, to wrap your head around object oriented programming, you may like implementing the tracer particle first, so that your diffusion code stays intact. Or you might want to wait with commenting until the end if you think you'll have to change a module for this assignment.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by
3:00 pm on Friday February 8, 2013.

===HW4===

In this homework, we are going to implement the class project of a tracer particle coupled to a diffusion equation.
The full specification of the physical problem is [[Media:ScClassProject.pdf|here]].
* Augment the tracer particle to include a force in the x and in the y direction, and a friction coefficient alpha, which at first can be constant.
* Implement the so-called leapfrog integration algorithm for the tracer particle
:::v ← v + f(v) Δt / m
:::r ← r + v Δt
:where v,r, and f are 2d vectors and f(v) is the total, velocity dependence force speficied in the class project, i.e., the sum of the external force F=qE and the friction force -αv. (Note: the v dependence of f make this strictly not a leapfrog integration, but we'll ignore that here.)
* Further augment the tracer class with a member function 'couple' which takes a diffusion field as input, and adjusts the friction constant.
* Your implementation of the 'couple' member function will need to interpolate the diffusion field to the current position of the particle. Use [[Media:CppInterpolation.tgz|this interpolation module]].
* Rewrite your main routine so that before tracer's time step, one calls the coupling. You may need to modify the Diffusion class a bit to get <tt>rho[active]</tt> out.
* For simplicity, use the same time step for both the diffusion and the tracer particle.
* Keep using git and make.

You will hand in your source code, makefiles and the git log file of all your commits by email by 9:00 am on Thursday February 21, 2013. Email the files, preferably zipped or tarred, to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

=Part 2: Numerical Tools for Physical Scientists=

==Prerequisites==

Part 1 or solid c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python (Enthought) installed on your laptop.

==Dates==

February 12, 14, 26, and 28, 2013 
March 5, 7, 12, and 14, 2013

==Topics==

===''Lecture 1:'' Numerics ===
:::[[File:Lecture9-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.html]]
:::[[Media:Lecture9-2013-Numerics.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.mp4 Video recording]

===''Lecture 2:'' Random numbers ===
:::[[File:Lecture10-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.html]]
:::[[Media:Lecture10-2013-PRNG.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW1_2 Homework assignment 1]

===''Lecture 3:'' Numerical integration and ODEs ===
:::[[File:Lecture11-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.html]]
:::[[Media:Lecture11-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.mp4 Video recording]

===''Lecture 4:'' Molecular Dynamics ===
:::[[File:Lecture12-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.html]]
:::[[Media:Lecture12-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW2_2 Homework assignment 2]

===''Lecture 5:'' Linear Algebra part I ===
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 6)]]

===''Lecture 6:'' Linear Algebra part II and PDEs===
:::[[File:Lecture14-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.html]]
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 5)]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW3_2 Homework assignment 3]

===''Lecture 7:'' Fast Fourier Transform===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.html]]
:::[[Media:Lecture15-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.mp4 Video recording]  /  [[Media:Sincfftw.cc|example code]]

===''Lecture 8:'' FFT for real and multidimensional data===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.html]]
:::[[Media:Lecture16-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.mp4 Video recording]  /   [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW4_2 Homework assignment 4]

==Homework Assignments==

===HW1===
This week's homework consists of two assignments.

''Assignment 1''

* Consider the sequence of numbers: 1 followed by 108 values of 10-8
* Should sum to 2
* Write code which sums up those values in order. What answer does it get?
* Add to program routine which sums up values in reverse order. Does it get correct answer?
* How would you get correct answer?
* Submit code, Makefile, text file with answers.

''Assignment 2''

* Implement an linear congruential generator with a = 106, c = 1283, m = 6075 that generates random numbers from 0..1
* Using that and MT: generate 10,000 pairs (dx, dy) with dx, dy each in -0.1 .. +0.1. Generate histograms of dx and dy (say 200 bins). Does it look okay? What would you expect variation to be?
* For 10,000 points: take random walks from (x,y)=(0,0) until exceed radius of 2, then stop. Plot histogram of final angles for the two psuedo random number generators. What do you see?
* Submit makefile, code, plots, git log.

Both assignments due on Thursday Feb 28th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW2===

''Assignment 1''

* Compute numerically (using the GSL):

::∫03 f(x)  dx

:(that is the integral of f(x) from x=0 to x=3)

:with

::f(x) = ln(x) sin(x) e-x

:using three different methods:
# Extended Simpsons' rule
# Gauss-Legendre quadrature
# Monte Carlo sampling

*Hint: what is f(0)?

*Compare the convergence of these methods by increasing number of function evaluations.

*Submit makefile, code, plots, version control log.

''Assignment 2''

* Using an adaptive 4th order Runge-Kutta approach, with a relative accuracy of 1e-4, compute the solution for t = [0,100] of the following set of coupled ODEs (Lorenz oscillator)

::dx/dt = σ(y - x)

::dy/dt = (ρ-z)x-y

::dz/dt = xy - βz

:with σ=10; β=8/3; ρ = 28, and with initial conditions

::x(0) = 10

::y(0) = 20

::z(0) = 30

* Hint: study the GSL documentation.

*Submit makefile, code, plots, version control log.

Both assignments due on Thursday Mar 7th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW3===

Part 1:

The time-explicit formulation of the 1d diffusion equation looks like this:
<math>
\begin{eqnarray*}
q^{n+1} & = & q^n + \frac{D \Delta t}{\Delta x^2}
\left (
\begin{matrix}
-2 & 1 \\
1 & -2 & 1 \\
& 1 & -2 & 1 \\
& & & \cdots & \\
& & & 1 & -2 & 1 \\
& & & & 1 & -2 \\
\end{matrix}
\right ) q^n \\
& = & \left ( 1 + \frac{D \Delta t}{\Delta x^2} A \right ) q^n
\end{eqnarray*}
</math>

what are the eignvalues of the matrix A? What modes would we expect to be amplified/damped by this operator?

* Consider 100 points in the discretization (eg, A is 100x100)
* Calculate the eigenvalues and eigenvectors (using D__EV ; which sort of matrix are we using here?)
* Plot the modes with the largest and smallest absolute-value of eigenvalues, and explain their physical significance
* The numerical method will become unstable with one eigenmode $v$ begins to grow uncontrollably whenever it is present, e.g.
$ \frac{D \Delta t}{\Delta x^2} A v = \frac{D \Delta t}{\Delta x^2} \lambda v > v$. In a timestepping solution, the only way to avoid this for a given physical set of parameters and grid size is to reduce the timestep, $\Delta t$. Use the largest absolute value eigenvalue to place a constraint on $\Delta t$ for stability.

Part 2:

Using the above constraint on $\Delta t$, for a 1d grid of size 100 (eg, a 100x100 matrix A), using lapack, evolve this PDE. Plot and explain results.

* Have an initial condition of $q(x=0,t=0) = 1$, and $q(t=0)$ everywhere else being zero (eg, hot plate just turned on at the left)
* Take ~100 timesteps and plot the the evolution of $q(x,t)$ at 5 times over that period.
* You’ll want to use a matrix multiply to compute the matrix-vector multiply ( http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html). Do multiply in double precision (D__MV). Which should you use?
* The GSL has a cblas interface, http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html ; an example of its use can be found here http://www.gnu.org/software/gsl/manual/html_node/GSL-CBLAS-Examples.html

Important things to know about lapack:
* If you are using an nxn array, the “leading dimension” of the array is n. (This argument is so that you could work on sub-matrices if you wanted)
* Have to make sure the 2d array is contiguous block of memory
* You'll (presumably) want to use the C bindings for LAPACK - [http://www.netlib.org/lapack/lapacke.html lapacke]. Note that the usual C arrays are row-major.

Here's a simple example of calling a LAPACKE routine; note that how the matrix is described (here with a pointer to the data, a leading dimension, and the number of rows and columns) will vary with different types of matrix:

<source lang="cpp">
#include <iostream>
#include <mkl_lapacke.h>

double **matrix(int n,int m);
void free_matrix(double **a);

int main (int argc, const char * argv[])
{

const int n=5; // number of rows, columns of the matrix
const int m = n; // nrows
const int leading_dim_A=n; // leading dimension (# of cols for row major);
// lets us operate on sub-matrices in principle
const int leading_dim_b=n; // similarly for b
double **A;
double *b;

b = new double[leading_dim_b];
A = matrix(n,leading_dim_A);

for (int i=0; i<n; i++)
for (int j=0; j<leading_dim_A; j++)
A[i][j] = 0.;

// let's do a trivial solve
// It should be pretty clear that the solution to this system
// is x = {0,1,2...n-1}

for (int i=0; i<leading_dim_A; i++) {
A[i][i] = 2.;
}

for (int i=0; i<leading_dim_b; i++) {
b[i] = 2*i;
}

const char transpose='N'; //solve Ax=b, not A^T x = b
const int nrhs = 1; // we're only solving 1 right hand side
int info;

// Call DGELS; b will be overwritten with the value of x.
info = LAPACKE_dgels(LAPACK_COL_MAJOR,transpose,m,n,nrhs,
&(A[0][0]),leading_dim_A, &(b[0]),leading_dim_b);

// print results
for(int i=0;i<n;i++)
{
if (i != n/2)
std::cout << " " << b[i] << std::endl;
else
std::cout << "x = " << b[i] << std::endl;
}
return(info);
}

double **matrix(int n,int m) {
double **a = new double * [n];
a[0] = new double [n*m];

for (int i=1; i<n; i++)
a[i] = &a[0][i*m];

return a;
}

void free_matrix(double **a) {
delete[] a[0];
delete[] a;
}
</source>

===HW4===

''Assignment 1''

Trigonometric interpolation uses a n point Fourier series to find values at intermediate points. It is one way of downscaling data, and was a motivation for Gauss, to be applied to planetary motion.

The way it works is:

# You fourier-transform your data
# You add frequecies above the Nyquist frequency (in absolute values), but set all the amplitudes of the new frequencies to zero.
# Note that the frequencies are stored such that eg. fn-1 is a low frequency -1.
# The resulting 2n array can be back transformed, and now gives an interpolated signal.

For this assignment, write an application that will read in an image from a binary file into a 2d double precision array (this will require converting from bytes to doubles), and creates an image twice the size in all directions using trigonometric interpolation. Use a real-to-half-complex version of the fftw (note: in 2d, this version of the fftw mixes fourier components with the same physical magnitude of their wave number k, so this will work).
You can process the red, green and blue values separately.

''Assignment 2''

Write an application which reads an image and performs a low pass filter on the image, i.e., any fourier components with magnitudes k larger than n/8 are to be set to zero, after which the fourier inverse is taken and the image is written out to disk again. Use the same fft technique as in the first assignment.

'''Input image'''

Use [[Media:gauss256.tgz|this image of Gauss]].

'''Image format:'''

Use the following simple PPM format:

First line (ascii): 'P6\n' 
Second line, in ascii, 'width height\n' 
Third line, in ascii, 'maxcolorvalue\n' (this is typically just 255) 
Following that, in binary, are byte-triplets with the red, green and blue values of each pixel. 
Note: in C, the 'unsigned char' data type matches the concept of a byte best (for most machines anyway).

In fact, between the first and second line, one can have comment lines that start with '#'.

=Part 3: High Performance Scientific Computing=

==Prerequisites==

Part 1 or good c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

You will need to bring a laptop with a ssh facility. Hands-on parts will be done on SciNet's GPC cluster.

For those who don't have a SciNet account yet, the instructions can be found at http://wiki.scinethpc.ca/wiki/index.php/Essentials#Accounts

==Dates==
March 19, 21, 26, and 28, 2013 
April 2, 4, 9, and 11, 2013

==Topics==
===''Lecture 1:'' Introduction to Parallel Programming ===
:::[[File:Lecture17-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.html]]
:::[[Media:Lecture17-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.mp4 Video recording]

===''Lecture 2:'' Parallel Computing Paradigms ===

:::[[File:Lecture18-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.html]]
:::[[Media:Lecture18-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.mp4 Video recording]  /   [[#HW1_3|homework 1]]

===''Lectures 3,4:'' Shared Memory Programming with OpenMP, part 1,2===

:::[[Media:Lecture19-2013.pdf|Slides]]

===''Lecture 5:'' Distributed Parallel Programming with MPI, part 1===

:::[[Media:Lecture21-2013.pdf|Slides]]

===''Lecture 6:'' Distributed Parallel Programming with MPI, part 2===

:::[[Media:Lecture22-2013.pdf|Slides]]

===''Lecture 7''   Distributed Parallel Programming with MPI, part 3===

:::[[Media:Lecture23-2013.pdf|Slides]]

''Lecture 8''   Hybrid OpenMPI+MPI Programming

== Homework assignments ==

=== HW1 ===

* Read the SciNet tutorial (as it pertains to the GPC)
* Read the GPC Quick Start.
* Get the first set of code:
<source lang="bash">
$ cd $SCRATCH
$ git clone /scinet/course/sc3/homework1
$ cd homework1
$ source setup
$ make
</source>
*This contains threaded program 'blurppm' and 266 ppm images to be blurred. Usage:
<source lang="bash">
blurppm INPUTPPM OUTPUTPPM BLURRADIUS NUMBEROFTHREADS
</source>
* Simple test:
<source lang="bash">
$ qsub -l nodes=1:ppn=8,walltime=2:00:00 -I -X -qdebug
$ cd $SCRATCH/homework1
$ time blurppm 001.ppm new001.ppm 30 1
real 0m52.900s
user 0m52.881s
sys 0m0.008s
$ display 001.ppm &
$ display new001.ppm &
</source>
''Assignment 1''
* Time blurppm with a BLURRADIUS ranging from 1 to 41 in steps of 4, and for NUMBEROFTHREADS ranging from 1 to 16. Record the (real) duration of each run.
* Plot the duration as a function of NUMBEROFTHREADS, as well as the speed-up and efficiency.
* Submit script and plots of the duration, speedup and effiency as a function of NUMBEROFTHREADS.
''Assignment 2''
* Use GNU parallel to run blurppm on all 266 images with a radius of 41.
* Investigate different scenarios:
:# Have GNU parallel run 16 at a time with just 1 thread.
:# Have GNU parallel run 8 at a time with 2 threads.
:# Have GNU parallel run 4 at a time with 4 threads.
:# Have GNU parallel run 2 at a time with 8 threads.
:# Have GNU parallel run 1 at a time with 16 threads.
:Record the total time it takes in each of these scenarios.
* Repeat this with a BLURRADIUS of 3.
* Submit scripts, timing data and plots.

=== HW2 ===

In the course materials ( /scinet/course/ppp/nbodyc or nbodyf ) there is the source code for a serial N-body integrator. This, like the molecular dynamics you've seen earlier, calculates long-range forces by particles on all of the other particles.

OpenMP the force calculation, and present timing results for 1,4, and 8 threads compared to the serial version. Note that you can turn off graphic output by removing the "USEPGPLOT = -DPGPLOT" line in Makefile.inc in the top level directory.

Begin by doubling the work by _not_ calculating two forces at once (eg, not making use of fji = -fij), and simply parallelizing the outer force loop. Then find a way to implement the forces efficiently but also in parallel. Is there any other part of the problem which could usefully be parallelized?

=== HW3 ===

In the same area ( /scinet/course/ppp/diffusion/diffusion.c ) there is a 1d diffusion problem of a form you may recognize from earlier modules. Your task is to parallelize this with MPI. I'd encourage you to use the graphics output while you're developing this (use -DPGPLOT on the compile line), and then omit the graphics and do timings with a larger totpoints (say 16000) and run timings on 1, 2, 4, and 8 processors. Note that for this simple problem, we don't necessarily expect a huge speedup; it's already pretty fast.

I'd suggest doing this in steps:

* include mpi.h, add mpi_init/finalize/comm_size/comm_rank, and compile with mpicc and make sure it runs;
* calculate the local number of points from the total number of points and the size (and possibly rank); treat the case where the total number of points isn't divisible by the number of processors however you like, but make sure you're consistent about it.
* Once you've calculated the local number of points, you won't need the variable totpoints; no arrays will be declared of that size, no plots will be made of that size, etc. Make those changes, compile, and run on (say) 2 procs.
* Now you'll find that you'll need to figure out the local xleft and local xright of the domain; again, once this is done you won't need to know the global variables any more. Make those changes, compile, and run.
* Finally, after the "old" boundary condition setting, do the internal boundary conditions, as in our example on class on Tuesday with sending messages around to our neighbour.

=Links=

==Unix==
* Cygwin: http://www.cygwin.com
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
* Intro to unix shell from software carpentry: http://software-carpentry.org/4_0/shell

==C/C++==
* [[One-Day Scientific C++ Class]] at SciNet
* C++ library reference: http://www.cplusplus.com/reference
* C preprocessor: http://www.cprogramming.com/tutorial/cpreprocessor.html
* Boost: http://www.boost.org
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==Git==
* Git: http://git-scm.com
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
* Git cheat sheet from Git Tower: http://www.git-tower.com/files/cheatsheet/Git_Cheat_Sheet_grey.pdf

==Python==
* Python: http://www.python.org
* IPython: http://ipython.org
* Matplotlib: http://www.matplotlib.org
* Enthought python distribution: http://www.enthought.com/products/edudownload.php 
(this gives you numpy, matplotlib and ipython all installed in one fell swoop)

* Intro to python from software carpentry: http://software-carpentry.org/4_0/python
* Tutorial on matplotlib: http://conference.scipy.org/scipy2011/tutorials.php#jonathan
* Npy file format: https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==ODEs==
* Integrators for particle based ODEs (i.e. molecular dynamics): http://www.chem.utoronto.ca/~rzon/simcourse/partmd.pdf. '''Focus on 4.1.4 - 4.1.6 for practical aspects.'''
* Numerical algorithm to solve ODEs (General) in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Chapter 16

==Interpolation (2D) ==
* Interpolation in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Pages 123-128
* Wikipedia pages on [http://en.wikipedia.org/wiki/Bilinear_interpolation Bilinear Interpolation] and [http://en.wikipedia.org/wiki/Bicubic_interpolation Bicubic Interpolation] are not bad either.

==BLAS==
* [http://www.tacc.utexas.edu/tacc-projects/gotoblas2 gotoblas]
* [http://math-atlas.sourceforge.net/ ATLAS]

==LAPACK==
* http://www.netlib.org/lapack

==GSL==
* GNU Scientific Library: http://www.gnu.org/s/gsl

==FFT==
* FFTW: http://www.fftw.org

==Top500==
* TOP500 Supercomputing Sites: http://top500.org

==OpenMP==
* OpenMP (open multi-processing) application programming interface for shared memory programming: http://openmp.org

==GNU parallel==
* Official citation: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

==SciNet==

Anything on this wiki, really, but specifically:
* [[Essentials|SciNet Essentials]]
* [[GPC Quickstart]]
* [[Media:SciNet_Tutorial.pdf |SciNet User Tutorial]]
* [[Software and Libraries]]

==Other Resources==
* [http://galileo.phys.virginia.edu/classes/551.jvn.fall01/goldberg.pdf What Every Computer Scientist Should Know About Floating-Point Arithmetic] - the classic (and extremely comprehensive) overview of the basics of floating point math. The first few pages, in particular, are very useful.
* [http://arxiv.org/abs/1005.4117 Random Numbers In Scientific Computing: An Introduction] by Katzgraber. A very lucid discussion of pseudo random number generators for science.

Scientific Computing Course

2013-04-05T20:23:32Z

Ljdursi: /* Homework assignments */

''This wiki page concerns the 2013 installment of SciNet's Scientific Computing course. Material from the previous installment can be found on [[Scientific Software Development Course]], [[Numerical Tools for Physical Scientists (course)]], and [[High Performance Scientific Computing]]''

__TOC__
=Syllabus=

==About the course==
* Whole-term graduate course
* Prerequisite: basic C, C++ or Fortran experience.
* Will use `C++ light' and Python
* Topics include: Scientific computing and programming skills, Parallel programming, and Hybrid programming.

There are three parts to this course:

# Scientific Software Development: Jan/Feb 2013 ''python, C++, git, make, modular programming, debugging''
# Numerical Tools for Physical Scientists: Feb/Mar 2013 ''modelling, floating point, Monte Carlo, ODE, linear algebra,fft''
# High Performance Scientific Computing: Mar/Apr 2013 ''openmp, mpi and hybrid programming''

Each part consists of eight one-hour lectures, two per week.

These can be taken separately by astrophysics graduate students at the University of Toronto as mini-courses, and by physics graduate students at the University of Toronto as modular courses.

The first two parts count towards the SciNet Certificate in Scientific Computing, while the third part can count towards the SciNet HPC Certificate. For more info about the SciNet Certificates, see http://www.scinethpc.ca/2012/12/scinet-hpc-certificate-program.

==Location and Times==
[http://www.scinethpc.ca/2010/08/contact-us SciNet HeadQuarters] 
256 McCaul Street, Toronto, ON 
Room 229 (Conference Room) 
Tuesdays 11:00 am - 12:00 noon 
Thursdays 11:00 am - 12:00 noon

==Instructors and office hours==

* Ramses van Zon - 256 McCaul Street, Rm 228 - Mondays 3-4pm
* L. Jonathan Dursi - 256 McCaul Street, Rm 216 - Wednesdays 3-4pm

==Grading scheme==

Attendence to lectures.

Four home work sets (i.e., one per week), to be returned by email by 9:00 am the next Thursday.

==Sign up==
Sign up for this graduate course goes through SciNet's course website. The direct link is https://support.scinet.utoronto.ca/courses/?q=node/99. If you do not have a SciNet account but wish to register for this course, please email support@scinet.utoronto.ca . 
Sign up is closed.

=Part 1: Scientific Software Development=

==Prerequisites==

Some programming experience. Some unix prompt experience.

'''Software that you'll need:'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python 2, IPython, Numpy, SciPy and Matplotlib (which you all get if you use the Enthought distribution) installed on your laptop. Links are given at the bottom of this page.

==Dates==

January 15, 17, 22, 24, 29, and 31, 2013 
February 5 and 7, 2013

==Topics (with lecture slides and recordings)==

===''Lecture 1:'' C++ introduction===
:::[[File:Lecture1-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.html]]
:::[[Media:Lecture1-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.mp4 Video recording]

===''Lecture 2:'' More C++, build and version control ===
:::[[File:Lecture2-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.html]]
:::Guest lecturer: Michael Nolta (CITA) for the git portion of the lecture.
:::[[Media:Lecture2-2013.pdf|C++ and Make slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.mp4 C++ and Make video recording]  /   [[Media:Git-Nolta.pdf|Git slides]]  /   [[#HW1|Homework assigment 1]]

===''Lecture 3:'' Python and visualization===
:::[[File:Lecture3-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.html]]
:::[[Media:Lecture3-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.mp4 Video recording]

===''Lecture 4:'' Modular programming, refactoring, testing===
:::[[File:Lecture4-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.html]]
:::[[Media:Lecture4-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.mp4 Video recording]  /   [[#HW2|Homework assigment 2]]
:::[http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc (course project source file)]  /   [http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py (corresponding python movie generator)]

===''Lecture 5:'' Object oriented programming===
:::[[Media:Lecture5-2013.pdf|Slides]]
:::Recordings of this lecture are missing, but you could view the videos of SciNet's [[One-Day Scientific C++ Class]], in particular the parts on classes, polymorphism, and inheritance.

===''Lecture 6:'' ODE, interpolation===
:::[[File:Lecture6-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.html]]
:::[[Media:ScientificComputing2013-Lecture5-ODE.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.mp4 Video recording]  /   [[#HW3|Homework assigment 3]]

===''Lecture 7:'' Development tools: debugging and profiling===
:::[[File:Lecture7-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.html]]
:::[[Media:ScientificComputing2013-Debugging.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.mp4 Video recording]

===''Lecture 8:'' Objects in Python, linking C++ and Python===
:::[[File:Lecture8-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.html]]
:::[[Media:Lecture8-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.mp4 Video recording]

==Homework assignments==

===HW1===

'''''Multi-file C++ program to create a data file'''''

We’ve learned programming in basic C++, use of make and Makefiles to build projects, and local use of git for version control. In this first assignment, you’ll use these to make a multi-file C++ program, built with make, which computes and outputs a data file.

* Start a git repository, and begin writing a C++ program to
:# Get an array size and a standard deviation from user input,
:# Allocate a 2d array (use the code given in lecture 2),
:# Store a 2d Gaussian with a maximum at the centre of the array and given standard deviation (in units of grid points),
:# Output that array to a text file,
:# Free the array, and exit.
* The output text file should contain just the data in text format, with a row of the file corresponding to a row of the array and with whitespace between the numbers.
* The 2d array creation/freeing routines should be in one file (with an associated header file), the gaussian calculation be in another (ditto), and the output routine be in a third, with the main program calling each of these.
* Use a makefile to build your code (add it to the repository).
* You can start with everything in one file, with hardcoded values for sizes and standard deviation and a static array, then refactor things into multiple files, adding the other features.
* As a test, use the ipython executable that came with your Enthought python distribution to read your data and plot it. If your data file is named ‘data.txt’, running the following:
<pre>
$ ipython --pylab
In [1]: data = numpy.genfromtxt('data.txt')
In [2]: contour(data)
</pre>
should give a nice contour plot of a 2-dimensional gaussian.
* Email in your source code, makefile and the "git log" output of all your commits by email by at 9:00 am Thursday Jan 24th, 2013. Please zip or tar these files together as one attachment, with a file name that includes your name and "HW1".

===HW2===
'''''Refactor legacy code to a modular project with unit tests'''''

In class, today, we talked about modular programming and testing, and the project we’ll be working on for the next three weeks. This homework will start advancing on that project by working on the “legacy” code given to us by our supervisor ([http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc]), with a corresponding python plotting script ([http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py]), and whipping it into shape before we start adding new physics.
* Start a git repository for this project, and add the two files.
* Create a Makefile and add it to the repository.
* Since we have no tests, run the program with console output redirected to a file:
:<pre>$ diffuse > original-output.txt</pre>
 ''It turns out the code has a bug that can make the output different when the same code is run again, which obviously would not be good for a baseline test. Replace 'float error;' by 'float error=0.0;' to fix this.''
* Also save the two .npy output files, e.g. to original-data.npy and original-theory.npy. The triplet of files (original-output.txt, original-data.npy and original-theory.npy) serve as a baseline integrated test (add these to repository).
* Then write a 'test' target in your makefile that:
** Runs 'diffuse' with output to a new file.
** Compares the file with the baseline test file, and compare the .npy files.
:: (hint: the unix command diff or cmp can compare files).
* First refactoring: Move the global variables into the main routine.
* ''Chorus: Test your modified code, and commit.''
* Second refactoring: Extract a diffusion operator routine, that gets called from main.
* ''Chorus''
* Create a .cc/.h module for the diffusion operator.
* ''Chorus''
* Add two tests for the diffusion operator: for a constant and for a linear input field (<tt>rho[i][j]=a*i+b*j</tt>). Add these to the test target in the makefile.
* ''Chorus''
* More refactoring: Extract three more .cc/.h modules:
** for output (should not contain hardcoded filenames)
** computation of the theory
** and for the array allocation stuff.
* ''Chorus''
* Describe, but don't implement in the .h and .cc, what would be appropriate unit tests for these three modules.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by 9:00 am on Thursday January 31, 2013.

===HW3===
This week, we learned about object oriented programming, which fits nicely within the modular programming idea. In this homework, we are going to use some of it to restructure our code and get it ready to add the tracer particle, the goal of the course project.

The goal will be to have an instance of a <tt>Diffusion</tt> class,
as well as an instance of <tt>Tracer</tt>, which for now will be a
free particle moving as ('''x'''(t),'''y'''(t)) = ('''x'''(0) +
'''vx''' t, '''y'''(0) + '''vy''' t), without any coupling yet (we
will handle this next week).

To be more specific:
* Clean up your code, using the feedback from your HW2 grading, such that the modules are as independent as possible.
* If you have not done so yet, add comments to the header files of your modules to explain exactly what each function does (without going into implementation details), what its arguments mean and what it returns (unless it's a void function, of course).
* Objectify the <tt>main</tt> routine, by creating a class <tt>Diffusion</tt>.
* Put this class in its own module (declaration in .h, implementation in .cc). For instance, the declaration could be
<pre>
// diffusion.h
#ifndef DIFFUSIONH
#define DIFFUSIONH
#include <fstream>
class Diffusion {
public:
Diffusion(float x1, float x2, float D, int numPoints);
void init(float a0, float sigma0); // set initial field
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
float getRho(int i, int j); // get a value of the field
~Diffusion();
private:
float*** rho;
...
};
#endif
</pre>
(this is not supposed to be prescriptive.)
* In the implementation file you'd have things like
<pre>
// diffusion.cc
#include "diffusion.h"
...
void Diffusion::timeStep(float dt)
{
// code for the timeStep ...
}
...
</pre>
(note the inclusion of the module's header file on the top of the implementation, so the class is declared).
* Let <tt>int main()</tt> have the same functionality as before, but now by defining the parameters of the run, creating an object of this class, setting up file streams, and taking time steps and writing out by using calls to member functions of this object.
* Additionally, write a class <tt>Tracer</tt> which for now implements a free particle in 2d. Something like:
<pre>
class Tracer {
public:
Tracer(float x1, float x2);
void init(float x0, float y0, float vx, float vy);
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
~Tracer();
private:
...
};
</pre>
:The timeStep implementation can in this case use the infamous forward Euler integration scheme, because it happens to be exact here.
:When it comes to output to a npy file, let's view the the data of the tracer particle at one point in time as a 2x2 matrix <tt>[[x,y],[vx,vy]]</tt>, so we can use much of the npy output code that we used for the diffusion field, which was a (numPoints+2)x(numPoints+2) matrix.
* This class too should be its own module (Often, "one class, one module" is a good paradigm, though occasionally you'll have closely related classes).
* Add some code to int main to have the Tracer particle evolve at the same time as the diffusion field (although the two are completely uncoupled).
* Keep using git and make, run the tests that you have regularly to make sure your program still works.

Note that because we've now set up our program in a modular fashion, you can do
different parts of this assignment in any order you want. For instance, to wrap your head around object oriented programming, you may like implementing the tracer particle first, so that your diffusion code stays intact. Or you might want to wait with commenting until the end if you think you'll have to change a module for this assignment.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by
3:00 pm on Friday February 8, 2013.

===HW4===

In this homework, we are going to implement the class project of a tracer particle coupled to a diffusion equation.
The full specification of the physical problem is [[Media:ScClassProject.pdf|here]].
* Augment the tracer particle to include a force in the x and in the y direction, and a friction coefficient alpha, which at first can be constant.
* Implement the so-called leapfrog integration algorithm for the tracer particle
:::v ← v + f(v) Δt / m
:::r ← r + v Δt
:where v,r, and f are 2d vectors and f(v) is the total, velocity dependence force speficied in the class project, i.e., the sum of the external force F=qE and the friction force -αv. (Note: the v dependence of f make this strictly not a leapfrog integration, but we'll ignore that here.)
* Further augment the tracer class with a member function 'couple' which takes a diffusion field as input, and adjusts the friction constant.
* Your implementation of the 'couple' member function will need to interpolate the diffusion field to the current position of the particle. Use [[Media:CppInterpolation.tgz|this interpolation module]].
* Rewrite your main routine so that before tracer's time step, one calls the coupling. You may need to modify the Diffusion class a bit to get <tt>rho[active]</tt> out.
* For simplicity, use the same time step for both the diffusion and the tracer particle.
* Keep using git and make.

You will hand in your source code, makefiles and the git log file of all your commits by email by 9:00 am on Thursday February 21, 2013. Email the files, preferably zipped or tarred, to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

=Part 2: Numerical Tools for Physical Scientists=

==Prerequisites==

Part 1 or solid c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python (Enthought) installed on your laptop.

==Dates==

February 12, 14, 26, and 28, 2013 
March 5, 7, 12, and 14, 2013

==Topics==

===''Lecture 1:'' Numerics ===
:::[[File:Lecture9-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.html]]
:::[[Media:Lecture9-2013-Numerics.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.mp4 Video recording]

===''Lecture 2:'' Random numbers ===
:::[[File:Lecture10-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.html]]
:::[[Media:Lecture10-2013-PRNG.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW1_2 Homework assignment 1]

===''Lecture 3:'' Numerical integration and ODEs ===
:::[[File:Lecture11-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.html]]
:::[[Media:Lecture11-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.mp4 Video recording]

===''Lecture 4:'' Molecular Dynamics ===
:::[[File:Lecture12-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.html]]
:::[[Media:Lecture12-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW2_2 Homework assignment 2]

===''Lecture 5:'' Linear Algebra part I ===
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 6)]]

===''Lecture 6:'' Linear Algebra part II and PDEs===
:::[[File:Lecture14-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.html]]
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 5)]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW3_2 Homework assignment 3]

===''Lecture 7:'' Fast Fourier Transform===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.html]]
:::[[Media:Lecture15-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.mp4 Video recording]  /  [[Media:Sincfftw.cc|example code]]

===''Lecture 8:'' FFT for real and multidimensional data===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.html]]
:::[[Media:Lecture16-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.mp4 Video recording]  /   [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW4_2 Homework assignment 4]

==Homework Assignments==

===HW1===
This week's homework consists of two assignments.

''Assignment 1''

* Consider the sequence of numbers: 1 followed by 108 values of 10-8
* Should sum to 2
* Write code which sums up those values in order. What answer does it get?
* Add to program routine which sums up values in reverse order. Does it get correct answer?
* How would you get correct answer?
* Submit code, Makefile, text file with answers.

''Assignment 2''

* Implement an linear congruential generator with a = 106, c = 1283, m = 6075 that generates random numbers from 0..1
* Using that and MT: generate 10,000 pairs (dx, dy) with dx, dy each in -0.1 .. +0.1. Generate histograms of dx and dy (say 200 bins). Does it look okay? What would you expect variation to be?
* For 10,000 points: take random walks from (x,y)=(0,0) until exceed radius of 2, then stop. Plot histogram of final angles for the two psuedo random number generators. What do you see?
* Submit makefile, code, plots, git log.

Both assignments due on Thursday Feb 28th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW2===

''Assignment 1''

* Compute numerically (using the GSL):

::∫03 f(x)  dx

:(that is the integral of f(x) from x=0 to x=3)

:with

::f(x) = ln(x) sin(x) e-x

:using three different methods:
# Extended Simpsons' rule
# Gauss-Legendre quadrature
# Monte Carlo sampling

*Hint: what is f(0)?

*Compare the convergence of these methods by increasing number of function evaluations.

*Submit makefile, code, plots, version control log.

''Assignment 2''

* Using an adaptive 4th order Runge-Kutta approach, with a relative accuracy of 1e-4, compute the solution for t = [0,100] of the following set of coupled ODEs (Lorenz oscillator)

::dx/dt = σ(y - x)

::dy/dt = (ρ-z)x-y

::dz/dt = xy - βz

:with σ=10; β=8/3; ρ = 28, and with initial conditions

::x(0) = 10

::y(0) = 20

::z(0) = 30

* Hint: study the GSL documentation.

*Submit makefile, code, plots, version control log.

Both assignments due on Thursday Mar 7th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW3===

Part 1:

The time-explicit formulation of the 1d diffusion equation looks like this:
<math>
\begin{eqnarray*}
q^{n+1} & = & q^n + \frac{D \Delta t}{\Delta x^2}
\left (
\begin{matrix}
-2 & 1 \\
1 & -2 & 1 \\
& 1 & -2 & 1 \\
& & & \cdots & \\
& & & 1 & -2 & 1 \\
& & & & 1 & -2 \\
\end{matrix}
\right ) q^n \\
& = & \left ( 1 + \frac{D \Delta t}{\Delta x^2} A \right ) q^n
\end{eqnarray*}
</math>

what are the eignvalues of the matrix A? What modes would we expect to be amplified/damped by this operator?

* Consider 100 points in the discretization (eg, A is 100x100)
* Calculate the eigenvalues and eigenvectors (using D__EV ; which sort of matrix are we using here?)
* Plot the modes with the largest and smallest absolute-value of eigenvalues, and explain their physical significance
* The numerical method will become unstable with one eigenmode $v$ begins to grow uncontrollably whenever it is present, e.g.
$ \frac{D \Delta t}{\Delta x^2} A v = \frac{D \Delta t}{\Delta x^2} \lambda v > v$. In a timestepping solution, the only way to avoid this for a given physical set of parameters and grid size is to reduce the timestep, $\Delta t$. Use the largest absolute value eigenvalue to place a constraint on $\Delta t$ for stability.

Part 2:

Using the above constraint on $\Delta t$, for a 1d grid of size 100 (eg, a 100x100 matrix A), using lapack, evolve this PDE. Plot and explain results.

* Have an initial condition of $q(x=0,t=0) = 1$, and $q(t=0)$ everywhere else being zero (eg, hot plate just turned on at the left)
* Take ~100 timesteps and plot the the evolution of $q(x,t)$ at 5 times over that period.
* You’ll want to use a matrix multiply to compute the matrix-vector multiply ( http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html). Do multiply in double precision (D__MV). Which should you use?
* The GSL has a cblas interface, http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html ; an example of its use can be found here http://www.gnu.org/software/gsl/manual/html_node/GSL-CBLAS-Examples.html

Important things to know about lapack:
* If you are using an nxn array, the “leading dimension” of the array is n. (This argument is so that you could work on sub-matrices if you wanted)
* Have to make sure the 2d array is contiguous block of memory
* You'll (presumably) want to use the C bindings for LAPACK - [http://www.netlib.org/lapack/lapacke.html lapacke]. Note that the usual C arrays are row-major.

Here's a simple example of calling a LAPACKE routine; note that how the matrix is described (here with a pointer to the data, a leading dimension, and the number of rows and columns) will vary with different types of matrix:

<source lang="cpp">
#include <iostream>
#include <mkl_lapacke.h>

double **matrix(int n,int m);
void free_matrix(double **a);

int main (int argc, const char * argv[])
{

const int n=5; // number of rows, columns of the matrix
const int m = n; // nrows
const int leading_dim_A=n; // leading dimension (# of cols for row major);
// lets us operate on sub-matrices in principle
const int leading_dim_b=n; // similarly for b
double **A;
double *b;

b = new double[leading_dim_b];
A = matrix(n,leading_dim_A);

for (int i=0; i<n; i++)
for (int j=0; j<leading_dim_A; j++)
A[i][j] = 0.;

// let's do a trivial solve
// It should be pretty clear that the solution to this system
// is x = {0,1,2...n-1}

for (int i=0; i<leading_dim_A; i++) {
A[i][i] = 2.;
}

for (int i=0; i<leading_dim_b; i++) {
b[i] = 2*i;
}

const char transpose='N'; //solve Ax=b, not A^T x = b
const int nrhs = 1; // we're only solving 1 right hand side
int info;

// Call DGELS; b will be overwritten with the value of x.
info = LAPACKE_dgels(LAPACK_COL_MAJOR,transpose,m,n,nrhs,
&(A[0][0]),leading_dim_A, &(b[0]),leading_dim_b);

// print results
for(int i=0;i<n;i++)
{
if (i != n/2)
std::cout << " " << b[i] << std::endl;
else
std::cout << "x = " << b[i] << std::endl;
}
return(info);
}

double **matrix(int n,int m) {
double **a = new double * [n];
a[0] = new double [n*m];

for (int i=1; i<n; i++)
a[i] = &a[0][i*m];

return a;
}

void free_matrix(double **a) {
delete[] a[0];
delete[] a;
}
</source>

===HW4===

''Assignment 1''

Trigonometric interpolation uses a n point Fourier series to find values at intermediate points. It is one way of downscaling data, and was a motivation for Gauss, to be applied to planetary motion.

The way it works is:

# You fourier-transform your data
# You add frequecies above the Nyquist frequency (in absolute values), but set all the amplitudes of the new frequencies to zero.
# Note that the frequencies are stored such that eg. fn-1 is a low frequency -1.
# The resulting 2n array can be back transformed, and now gives an interpolated signal.

For this assignment, write an application that will read in an image from a binary file into a 2d double precision array (this will require converting from bytes to doubles), and creates an image twice the size in all directions using trigonometric interpolation. Use a real-to-half-complex version of the fftw (note: in 2d, this version of the fftw mixes fourier components with the same physical magnitude of their wave number k, so this will work).
You can process the red, green and blue values separately.

''Assignment 2''

Write an application which reads an image and performs a low pass filter on the image, i.e., any fourier components with magnitudes k larger than n/8 are to be set to zero, after which the fourier inverse is taken and the image is written out to disk again. Use the same fft technique as in the first assignment.

'''Input image'''

Use [[Media:gauss256.tgz|this image of Gauss]].

'''Image format:'''

Use the following simple PPM format:

First line (ascii): 'P6\n' 
Second line, in ascii, 'width height\n' 
Third line, in ascii, 'maxcolorvalue\n' (this is typically just 255) 
Following that, in binary, are byte-triplets with the red, green and blue values of each pixel. 
Note: in C, the 'unsigned char' data type matches the concept of a byte best (for most machines anyway).

In fact, between the first and second line, one can have comment lines that start with '#'.

=Part 3: High Performance Scientific Computing=

==Prerequisites==

Part 1 or good c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

You will need to bring a laptop with a ssh facility. Hands-on parts will be done on SciNet's GPC cluster.

For those who don't have a SciNet account yet, the instructions can be found at http://wiki.scinethpc.ca/wiki/index.php/Essentials#Accounts

==Dates==
March 19, 21, 26, and 28, 2013 
April 2, 4, 9, and 11, 2013

==Topics==
===''Lecture 1:'' Introduction to Parallel Programming ===
:::[[File:Lecture17-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.html]]
:::[[Media:Lecture17-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.mp4 Video recording]

===''Lecture 2:'' Parallel Computing Paradigms ===

:::[[File:Lecture18-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.html]]
:::[[Media:Lecture18-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.mp4 Video recording]  /   [[#HW1_3|homework 1]]

===''Lectures 3,4:'' Shared Memory Programming with OpenMP, part 1,2===

:::[[Media:Lecture19-2013.pdf|Slides]]

===''Lecture 5:'' Distributed Parallel Programming with MPI, part 1===

:::[[Media:Lecture21-2013.pdf|Slides]]

===''Lecture 6:'' Distributed Parallel Programming with MPI, part 2===

:::[[Media:Lecture22-2013.pdf|Slides]]

''Lecture 7''   Distributed Parallel Programming with MPI, part 3 
''Lecture 8''   Hybrid OpenMPI+MPI Programming

== Homework assignments ==

=== HW1 ===

* Read the SciNet tutorial (as it pertains to the GPC)
* Read the GPC Quick Start.
* Get the first set of code:
<source lang="bash">
$ cd $SCRATCH
$ git clone /scinet/course/sc3/homework1
$ cd homework1
$ source setup
$ make
</source>
*This contains threaded program 'blurppm' and 266 ppm images to be blurred. Usage:
<source lang="bash">
blurppm INPUTPPM OUTPUTPPM BLURRADIUS NUMBEROFTHREADS
</source>
* Simple test:
<source lang="bash">
$ qsub -l nodes=1:ppn=8,walltime=2:00:00 -I -X -qdebug
$ cd $SCRATCH/homework1
$ time blurppm 001.ppm new001.ppm 30 1
real 0m52.900s
user 0m52.881s
sys 0m0.008s
$ display 001.ppm &
$ display new001.ppm &
</source>
''Assignment 1''
* Time blurppm with a BLURRADIUS ranging from 1 to 41 in steps of 4, and for NUMBEROFTHREADS ranging from 1 to 16. Record the (real) duration of each run.
* Plot the duration as a function of NUMBEROFTHREADS, as well as the speed-up and efficiency.
* Submit script and plots of the duration, speedup and effiency as a function of NUMBEROFTHREADS.
''Assignment 2''
* Use GNU parallel to run blurppm on all 266 images with a radius of 41.
* Investigate different scenarios:
:# Have GNU parallel run 16 at a time with just 1 thread.
:# Have GNU parallel run 8 at a time with 2 threads.
:# Have GNU parallel run 4 at a time with 4 threads.
:# Have GNU parallel run 2 at a time with 8 threads.
:# Have GNU parallel run 1 at a time with 16 threads.
:Record the total time it takes in each of these scenarios.
* Repeat this with a BLURRADIUS of 3.
* Submit scripts, timing data and plots.

=== HW2 ===

In the course materials ( /scinet/course/ppp/nbodyc or nbodyf ) there is the source code for a serial N-body integrator. This, like the molecular dynamics you've seen earlier, calculates long-range forces by particles on all of the other particles.

OpenMP the force calculation, and present timing results for 1,4, and 8 threads compared to the serial version. Note that you can turn off graphic output by removing the "USEPGPLOT = -DPGPLOT" line in Makefile.inc in the top level directory.

Begin by doubling the work by _not_ calculating two forces at once (eg, not making use of fji = -fij), and simply parallelizing the outer force loop. Then find a way to implement the forces efficiently but also in parallel. Is there any other part of the problem which could usefully be parallelized?

=== HW3 ===

In the same area ( /scinet/course/ppp/diffusion/diffusion.c ) there is a 1d diffusion problem of a form you may recognize from earlier modules. Your task is to parallelize this with MPI. I'd encourage you to use the graphics output while you're developing this (use -DPGPLOT on the compile line), and then omit the graphics and do timings with a larger totpoints (say 16000) and run timings on 1, 2, 4, and 8 processors. Note that for this simple problem, we don't necessarily expect a huge speedup; it's already pretty fast.

I'd suggest doing this in steps:

* include mpi.h, add mpi_init/finalize/comm_size/comm_rank, and compile with mpicc and make sure it runs;
* calculate the local number of points from the total number of points and the size (and possibly rank); treat the case where the total number of points isn't divisible by the number of processors however you like, but make sure you're consistent about it.
* Once you've calculated the local number of points, you won't need the variable totpoints; no arrays will be declared of that size, no plots will be made of that size, etc. Make those changes, compile, and run on (say) 2 procs.
* Now you'll find that you'll need to figure out the local xleft and local xright of the domain; again, once this is done you won't need to know the global variables any more. Make those changes, compile, and run.
* Finally, after the "old" boundary condition setting, do the internal boundary conditions, as in our example on class on Tuesday with sending messages around to our neighbour.

=Links=

==Unix==
* Cygwin: http://www.cygwin.com
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
* Intro to unix shell from software carpentry: http://software-carpentry.org/4_0/shell

==C/C++==
* [[One-Day Scientific C++ Class]] at SciNet
* C++ library reference: http://www.cplusplus.com/reference
* C preprocessor: http://www.cprogramming.com/tutorial/cpreprocessor.html
* Boost: http://www.boost.org
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==Git==
* Git: http://git-scm.com
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
* Git cheat sheet from Git Tower: http://www.git-tower.com/files/cheatsheet/Git_Cheat_Sheet_grey.pdf

==Python==
* Python: http://www.python.org
* IPython: http://ipython.org
* Matplotlib: http://www.matplotlib.org
* Enthought python distribution: http://www.enthought.com/products/edudownload.php 
(this gives you numpy, matplotlib and ipython all installed in one fell swoop)

* Intro to python from software carpentry: http://software-carpentry.org/4_0/python
* Tutorial on matplotlib: http://conference.scipy.org/scipy2011/tutorials.php#jonathan
* Npy file format: https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==ODEs==
* Integrators for particle based ODEs (i.e. molecular dynamics): http://www.chem.utoronto.ca/~rzon/simcourse/partmd.pdf. '''Focus on 4.1.4 - 4.1.6 for practical aspects.'''
* Numerical algorithm to solve ODEs (General) in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Chapter 16

==Interpolation (2D) ==
* Interpolation in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Pages 123-128
* Wikipedia pages on [http://en.wikipedia.org/wiki/Bilinear_interpolation Bilinear Interpolation] and [http://en.wikipedia.org/wiki/Bicubic_interpolation Bicubic Interpolation] are not bad either.

==BLAS==
* [http://www.tacc.utexas.edu/tacc-projects/gotoblas2 gotoblas]
* [http://math-atlas.sourceforge.net/ ATLAS]

==LAPACK==
* http://www.netlib.org/lapack

==GSL==
* GNU Scientific Library: http://www.gnu.org/s/gsl

==FFT==
* FFTW: http://www.fftw.org

==Top500==
* TOP500 Supercomputing Sites: http://top500.org

==OpenMP==
* OpenMP (open multi-processing) application programming interface for shared memory programming: http://openmp.org

==GNU parallel==
* Official citation: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

==SciNet==

Anything on this wiki, really, but specifically:
* [[Essentials|SciNet Essentials]]
* [[GPC Quickstart]]
* [[Media:SciNet_Tutorial.pdf |SciNet User Tutorial]]
* [[Software and Libraries]]

==Other Resources==
* [http://galileo.phys.virginia.edu/classes/551.jvn.fall01/goldberg.pdf What Every Computer Scientist Should Know About Floating-Point Arithmetic] - the classic (and extremely comprehensive) overview of the basics of floating point math. The first few pages, in particular, are very useful.
* [http://arxiv.org/abs/1005.4117 Random Numbers In Scientific Computing: An Introduction] by Katzgraber. A very lucid discussion of pseudo random number generators for science.

File:Lecture22-2013.pdf

2013-04-04T13:38:37Z

Ljdursi:

Scientific Computing Course

2013-04-04T13:38:12Z

Ljdursi: /* Lecture 5: Distributed Parallel Programming with MPI, part 1 */

''This wiki page concerns the 2013 installment of SciNet's Scientific Computing course. Material from the previous installment can be found on [[Scientific Software Development Course]], [[Numerical Tools for Physical Scientists (course)]], and [[High Performance Scientific Computing]]''

__TOC__
=Syllabus=

==About the course==
* Whole-term graduate course
* Prerequisite: basic C, C++ or Fortran experience.
* Will use `C++ light' and Python
* Topics include: Scientific computing and programming skills, Parallel programming, and Hybrid programming.

There are three parts to this course:

# Scientific Software Development: Jan/Feb 2013 ''python, C++, git, make, modular programming, debugging''
# Numerical Tools for Physical Scientists: Feb/Mar 2013 ''modelling, floating point, Monte Carlo, ODE, linear algebra,fft''
# High Performance Scientific Computing: Mar/Apr 2013 ''openmp, mpi and hybrid programming''

Each part consists of eight one-hour lectures, two per week.

These can be taken separately by astrophysics graduate students at the University of Toronto as mini-courses, and by physics graduate students at the University of Toronto as modular courses.

The first two parts count towards the SciNet Certificate in Scientific Computing, while the third part can count towards the SciNet HPC Certificate. For more info about the SciNet Certificates, see http://www.scinethpc.ca/2012/12/scinet-hpc-certificate-program.

==Location and Times==
[http://www.scinethpc.ca/2010/08/contact-us SciNet HeadQuarters] 
256 McCaul Street, Toronto, ON 
Room 229 (Conference Room) 
Tuesdays 11:00 am - 12:00 noon 
Thursdays 11:00 am - 12:00 noon

==Instructors and office hours==

* Ramses van Zon - 256 McCaul Street, Rm 228 - Mondays 3-4pm
* L. Jonathan Dursi - 256 McCaul Street, Rm 216 - Wednesdays 3-4pm

==Grading scheme==

Attendence to lectures.

Four home work sets (i.e., one per week), to be returned by email by 9:00 am the next Thursday.

==Sign up==
Sign up for this graduate course goes through SciNet's course website. The direct link is https://support.scinet.utoronto.ca/courses/?q=node/99. If you do not have a SciNet account but wish to register for this course, please email support@scinet.utoronto.ca . 
Sign up is closed.

=Part 1: Scientific Software Development=

==Prerequisites==

Some programming experience. Some unix prompt experience.

'''Software that you'll need:'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python 2, IPython, Numpy, SciPy and Matplotlib (which you all get if you use the Enthought distribution) installed on your laptop. Links are given at the bottom of this page.

==Dates==

January 15, 17, 22, 24, 29, and 31, 2013 
February 5 and 7, 2013

==Topics (with lecture slides and recordings)==

===''Lecture 1:'' C++ introduction===
:::[[File:Lecture1-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.html]]
:::[[Media:Lecture1-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.mp4 Video recording]

===''Lecture 2:'' More C++, build and version control ===
:::[[File:Lecture2-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.html]]
:::Guest lecturer: Michael Nolta (CITA) for the git portion of the lecture.
:::[[Media:Lecture2-2013.pdf|C++ and Make slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.mp4 C++ and Make video recording]  /   [[Media:Git-Nolta.pdf|Git slides]]  /   [[#HW1|Homework assigment 1]]

===''Lecture 3:'' Python and visualization===
:::[[File:Lecture3-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.html]]
:::[[Media:Lecture3-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.mp4 Video recording]

===''Lecture 4:'' Modular programming, refactoring, testing===
:::[[File:Lecture4-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.html]]
:::[[Media:Lecture4-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.mp4 Video recording]  /   [[#HW2|Homework assigment 2]]
:::[http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc (course project source file)]  /   [http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py (corresponding python movie generator)]

===''Lecture 5:'' Object oriented programming===
:::[[Media:Lecture5-2013.pdf|Slides]]
:::Recordings of this lecture are missing, but you could view the videos of SciNet's [[One-Day Scientific C++ Class]], in particular the parts on classes, polymorphism, and inheritance.

===''Lecture 6:'' ODE, interpolation===
:::[[File:Lecture6-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.html]]
:::[[Media:ScientificComputing2013-Lecture5-ODE.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.mp4 Video recording]  /   [[#HW3|Homework assigment 3]]

===''Lecture 7:'' Development tools: debugging and profiling===
:::[[File:Lecture7-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.html]]
:::[[Media:ScientificComputing2013-Debugging.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.mp4 Video recording]

===''Lecture 8:'' Objects in Python, linking C++ and Python===
:::[[File:Lecture8-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.html]]
:::[[Media:Lecture8-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.mp4 Video recording]

==Homework assignments==

===HW1===

'''''Multi-file C++ program to create a data file'''''

We’ve learned programming in basic C++, use of make and Makefiles to build projects, and local use of git for version control. In this first assignment, you’ll use these to make a multi-file C++ program, built with make, which computes and outputs a data file.

* Start a git repository, and begin writing a C++ program to
:# Get an array size and a standard deviation from user input,
:# Allocate a 2d array (use the code given in lecture 2),
:# Store a 2d Gaussian with a maximum at the centre of the array and given standard deviation (in units of grid points),
:# Output that array to a text file,
:# Free the array, and exit.
* The output text file should contain just the data in text format, with a row of the file corresponding to a row of the array and with whitespace between the numbers.
* The 2d array creation/freeing routines should be in one file (with an associated header file), the gaussian calculation be in another (ditto), and the output routine be in a third, with the main program calling each of these.
* Use a makefile to build your code (add it to the repository).
* You can start with everything in one file, with hardcoded values for sizes and standard deviation and a static array, then refactor things into multiple files, adding the other features.
* As a test, use the ipython executable that came with your Enthought python distribution to read your data and plot it. If your data file is named ‘data.txt’, running the following:
<pre>
$ ipython --pylab
In [1]: data = numpy.genfromtxt('data.txt')
In [2]: contour(data)
</pre>
should give a nice contour plot of a 2-dimensional gaussian.
* Email in your source code, makefile and the "git log" output of all your commits by email by at 9:00 am Thursday Jan 24th, 2013. Please zip or tar these files together as one attachment, with a file name that includes your name and "HW1".

===HW2===
'''''Refactor legacy code to a modular project with unit tests'''''

In class, today, we talked about modular programming and testing, and the project we’ll be working on for the next three weeks. This homework will start advancing on that project by working on the “legacy” code given to us by our supervisor ([http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc]), with a corresponding python plotting script ([http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py]), and whipping it into shape before we start adding new physics.
* Start a git repository for this project, and add the two files.
* Create a Makefile and add it to the repository.
* Since we have no tests, run the program with console output redirected to a file:
:<pre>$ diffuse > original-output.txt</pre>
 ''It turns out the code has a bug that can make the output different when the same code is run again, which obviously would not be good for a baseline test. Replace 'float error;' by 'float error=0.0;' to fix this.''
* Also save the two .npy output files, e.g. to original-data.npy and original-theory.npy. The triplet of files (original-output.txt, original-data.npy and original-theory.npy) serve as a baseline integrated test (add these to repository).
* Then write a 'test' target in your makefile that:
** Runs 'diffuse' with output to a new file.
** Compares the file with the baseline test file, and compare the .npy files.
:: (hint: the unix command diff or cmp can compare files).
* First refactoring: Move the global variables into the main routine.
* ''Chorus: Test your modified code, and commit.''
* Second refactoring: Extract a diffusion operator routine, that gets called from main.
* ''Chorus''
* Create a .cc/.h module for the diffusion operator.
* ''Chorus''
* Add two tests for the diffusion operator: for a constant and for a linear input field (<tt>rho[i][j]=a*i+b*j</tt>). Add these to the test target in the makefile.
* ''Chorus''
* More refactoring: Extract three more .cc/.h modules:
** for output (should not contain hardcoded filenames)
** computation of the theory
** and for the array allocation stuff.
* ''Chorus''
* Describe, but don't implement in the .h and .cc, what would be appropriate unit tests for these three modules.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by 9:00 am on Thursday January 31, 2013.

===HW3===
This week, we learned about object oriented programming, which fits nicely within the modular programming idea. In this homework, we are going to use some of it to restructure our code and get it ready to add the tracer particle, the goal of the course project.

The goal will be to have an instance of a <tt>Diffusion</tt> class,
as well as an instance of <tt>Tracer</tt>, which for now will be a
free particle moving as ('''x'''(t),'''y'''(t)) = ('''x'''(0) +
'''vx''' t, '''y'''(0) + '''vy''' t), without any coupling yet (we
will handle this next week).

To be more specific:
* Clean up your code, using the feedback from your HW2 grading, such that the modules are as independent as possible.
* If you have not done so yet, add comments to the header files of your modules to explain exactly what each function does (without going into implementation details), what its arguments mean and what it returns (unless it's a void function, of course).
* Objectify the <tt>main</tt> routine, by creating a class <tt>Diffusion</tt>.
* Put this class in its own module (declaration in .h, implementation in .cc). For instance, the declaration could be
<pre>
// diffusion.h
#ifndef DIFFUSIONH
#define DIFFUSIONH
#include <fstream>
class Diffusion {
public:
Diffusion(float x1, float x2, float D, int numPoints);
void init(float a0, float sigma0); // set initial field
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
float getRho(int i, int j); // get a value of the field
~Diffusion();
private:
float*** rho;
...
};
#endif
</pre>
(this is not supposed to be prescriptive.)
* In the implementation file you'd have things like
<pre>
// diffusion.cc
#include "diffusion.h"
...
void Diffusion::timeStep(float dt)
{
// code for the timeStep ...
}
...
</pre>
(note the inclusion of the module's header file on the top of the implementation, so the class is declared).
* Let <tt>int main()</tt> have the same functionality as before, but now by defining the parameters of the run, creating an object of this class, setting up file streams, and taking time steps and writing out by using calls to member functions of this object.
* Additionally, write a class <tt>Tracer</tt> which for now implements a free particle in 2d. Something like:
<pre>
class Tracer {
public:
Tracer(float x1, float x2);
void init(float x0, float y0, float vx, float vy);
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
~Tracer();
private:
...
};
</pre>
:The timeStep implementation can in this case use the infamous forward Euler integration scheme, because it happens to be exact here.
:When it comes to output to a npy file, let's view the the data of the tracer particle at one point in time as a 2x2 matrix <tt>[[x,y],[vx,vy]]</tt>, so we can use much of the npy output code that we used for the diffusion field, which was a (numPoints+2)x(numPoints+2) matrix.
* This class too should be its own module (Often, "one class, one module" is a good paradigm, though occasionally you'll have closely related classes).
* Add some code to int main to have the Tracer particle evolve at the same time as the diffusion field (although the two are completely uncoupled).
* Keep using git and make, run the tests that you have regularly to make sure your program still works.

Note that because we've now set up our program in a modular fashion, you can do
different parts of this assignment in any order you want. For instance, to wrap your head around object oriented programming, you may like implementing the tracer particle first, so that your diffusion code stays intact. Or you might want to wait with commenting until the end if you think you'll have to change a module for this assignment.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by
3:00 pm on Friday February 8, 2013.

===HW4===

In this homework, we are going to implement the class project of a tracer particle coupled to a diffusion equation.
The full specification of the physical problem is [[Media:ScClassProject.pdf|here]].
* Augment the tracer particle to include a force in the x and in the y direction, and a friction coefficient alpha, which at first can be constant.
* Implement the so-called leapfrog integration algorithm for the tracer particle
:::v ← v + f(v) Δt / m
:::r ← r + v Δt
:where v,r, and f are 2d vectors and f(v) is the total, velocity dependence force speficied in the class project, i.e., the sum of the external force F=qE and the friction force -αv. (Note: the v dependence of f make this strictly not a leapfrog integration, but we'll ignore that here.)
* Further augment the tracer class with a member function 'couple' which takes a diffusion field as input, and adjusts the friction constant.
* Your implementation of the 'couple' member function will need to interpolate the diffusion field to the current position of the particle. Use [[Media:CppInterpolation.tgz|this interpolation module]].
* Rewrite your main routine so that before tracer's time step, one calls the coupling. You may need to modify the Diffusion class a bit to get <tt>rho[active]</tt> out.
* For simplicity, use the same time step for both the diffusion and the tracer particle.
* Keep using git and make.

You will hand in your source code, makefiles and the git log file of all your commits by email by 9:00 am on Thursday February 21, 2013. Email the files, preferably zipped or tarred, to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

=Part 2: Numerical Tools for Physical Scientists=

==Prerequisites==

Part 1 or solid c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python (Enthought) installed on your laptop.

==Dates==

February 12, 14, 26, and 28, 2013 
March 5, 7, 12, and 14, 2013

==Topics==

===''Lecture 1:'' Numerics ===
:::[[File:Lecture9-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.html]]
:::[[Media:Lecture9-2013-Numerics.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.mp4 Video recording]

===''Lecture 2:'' Random numbers ===
:::[[File:Lecture10-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.html]]
:::[[Media:Lecture10-2013-PRNG.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW1_2 Homework assignment 1]

===''Lecture 3:'' Numerical integration and ODEs ===
:::[[File:Lecture11-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.html]]
:::[[Media:Lecture11-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.mp4 Video recording]

===''Lecture 4:'' Molecular Dynamics ===
:::[[File:Lecture12-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.html]]
:::[[Media:Lecture12-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW2_2 Homework assignment 2]

===''Lecture 5:'' Linear Algebra part I ===
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 6)]]

===''Lecture 6:'' Linear Algebra part II and PDEs===
:::[[File:Lecture14-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.html]]
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 5)]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW3_2 Homework assignment 3]

===''Lecture 7:'' Fast Fourier Transform===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.html]]
:::[[Media:Lecture15-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.mp4 Video recording]  /  [[Media:Sincfftw.cc|example code]]

===''Lecture 8:'' FFT for real and multidimensional data===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.html]]
:::[[Media:Lecture16-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.mp4 Video recording]  /   [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW4_2 Homework assignment 4]

==Homework Assignments==

===HW1===
This week's homework consists of two assignments.

''Assignment 1''

* Consider the sequence of numbers: 1 followed by 108 values of 10-8
* Should sum to 2
* Write code which sums up those values in order. What answer does it get?
* Add to program routine which sums up values in reverse order. Does it get correct answer?
* How would you get correct answer?
* Submit code, Makefile, text file with answers.

''Assignment 2''

* Implement an linear congruential generator with a = 106, c = 1283, m = 6075 that generates random numbers from 0..1
* Using that and MT: generate 10,000 pairs (dx, dy) with dx, dy each in -0.1 .. +0.1. Generate histograms of dx and dy (say 200 bins). Does it look okay? What would you expect variation to be?
* For 10,000 points: take random walks from (x,y)=(0,0) until exceed radius of 2, then stop. Plot histogram of final angles for the two psuedo random number generators. What do you see?
* Submit makefile, code, plots, git log.

Both assignments due on Thursday Feb 28th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW2===

''Assignment 1''

* Compute numerically (using the GSL):

::∫03 f(x)  dx

:(that is the integral of f(x) from x=0 to x=3)

:with

::f(x) = ln(x) sin(x) e-x

:using three different methods:
# Extended Simpsons' rule
# Gauss-Legendre quadrature
# Monte Carlo sampling

*Hint: what is f(0)?

*Compare the convergence of these methods by increasing number of function evaluations.

*Submit makefile, code, plots, version control log.

''Assignment 2''

* Using an adaptive 4th order Runge-Kutta approach, with a relative accuracy of 1e-4, compute the solution for t = [0,100] of the following set of coupled ODEs (Lorenz oscillator)

::dx/dt = σ(y - x)

::dy/dt = (ρ-z)x-y

::dz/dt = xy - βz

:with σ=10; β=8/3; ρ = 28, and with initial conditions

::x(0) = 10

::y(0) = 20

::z(0) = 30

* Hint: study the GSL documentation.

*Submit makefile, code, plots, version control log.

Both assignments due on Thursday Mar 7th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW3===

Part 1:

The time-explicit formulation of the 1d diffusion equation looks like this:
<math>
\begin{eqnarray*}
q^{n+1} & = & q^n + \frac{D \Delta t}{\Delta x^2}
\left (
\begin{matrix}
-2 & 1 \\
1 & -2 & 1 \\
& 1 & -2 & 1 \\
& & & \cdots & \\
& & & 1 & -2 & 1 \\
& & & & 1 & -2 \\
\end{matrix}
\right ) q^n \\
& = & \left ( 1 + \frac{D \Delta t}{\Delta x^2} A \right ) q^n
\end{eqnarray*}
</math>

what are the eignvalues of the matrix A? What modes would we expect to be amplified/damped by this operator?

* Consider 100 points in the discretization (eg, A is 100x100)
* Calculate the eigenvalues and eigenvectors (using D__EV ; which sort of matrix are we using here?)
* Plot the modes with the largest and smallest absolute-value of eigenvalues, and explain their physical significance
* The numerical method will become unstable with one eigenmode $v$ begins to grow uncontrollably whenever it is present, e.g.
$ \frac{D \Delta t}{\Delta x^2} A v = \frac{D \Delta t}{\Delta x^2} \lambda v > v$. In a timestepping solution, the only way to avoid this for a given physical set of parameters and grid size is to reduce the timestep, $\Delta t$. Use the largest absolute value eigenvalue to place a constraint on $\Delta t$ for stability.

Part 2:

Using the above constraint on $\Delta t$, for a 1d grid of size 100 (eg, a 100x100 matrix A), using lapack, evolve this PDE. Plot and explain results.

* Have an initial condition of $q(x=0,t=0) = 1$, and $q(t=0)$ everywhere else being zero (eg, hot plate just turned on at the left)
* Take ~100 timesteps and plot the the evolution of $q(x,t)$ at 5 times over that period.
* You’ll want to use a matrix multiply to compute the matrix-vector multiply ( http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html). Do multiply in double precision (D__MV). Which should you use?
* The GSL has a cblas interface, http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html ; an example of its use can be found here http://www.gnu.org/software/gsl/manual/html_node/GSL-CBLAS-Examples.html

Important things to know about lapack:
* If you are using an nxn array, the “leading dimension” of the array is n. (This argument is so that you could work on sub-matrices if you wanted)
* Have to make sure the 2d array is contiguous block of memory
* You'll (presumably) want to use the C bindings for LAPACK - [http://www.netlib.org/lapack/lapacke.html lapacke]. Note that the usual C arrays are row-major.

Here's a simple example of calling a LAPACKE routine; note that how the matrix is described (here with a pointer to the data, a leading dimension, and the number of rows and columns) will vary with different types of matrix:

<source lang="cpp">
#include <iostream>
#include <mkl_lapacke.h>

double **matrix(int n,int m);
void free_matrix(double **a);

int main (int argc, const char * argv[])
{

const int n=5; // number of rows, columns of the matrix
const int m = n; // nrows
const int leading_dim_A=n; // leading dimension (# of cols for row major);
// lets us operate on sub-matrices in principle
const int leading_dim_b=n; // similarly for b
double **A;
double *b;

b = new double[leading_dim_b];
A = matrix(n,leading_dim_A);

for (int i=0; i<n; i++)
for (int j=0; j<leading_dim_A; j++)
A[i][j] = 0.;

// let's do a trivial solve
// It should be pretty clear that the solution to this system
// is x = {0,1,2...n-1}

for (int i=0; i<leading_dim_A; i++) {
A[i][i] = 2.;
}

for (int i=0; i<leading_dim_b; i++) {
b[i] = 2*i;
}

const char transpose='N'; //solve Ax=b, not A^T x = b
const int nrhs = 1; // we're only solving 1 right hand side
int info;

// Call DGELS; b will be overwritten with the value of x.
info = LAPACKE_dgels(LAPACK_COL_MAJOR,transpose,m,n,nrhs,
&(A[0][0]),leading_dim_A, &(b[0]),leading_dim_b);

// print results
for(int i=0;i<n;i++)
{
if (i != n/2)
std::cout << " " << b[i] << std::endl;
else
std::cout << "x = " << b[i] << std::endl;
}
return(info);
}

double **matrix(int n,int m) {
double **a = new double * [n];
a[0] = new double [n*m];

for (int i=1; i<n; i++)
a[i] = &a[0][i*m];

return a;
}

void free_matrix(double **a) {
delete[] a[0];
delete[] a;
}
</source>

===HW4===

''Assignment 1''

Trigonometric interpolation uses a n point Fourier series to find values at intermediate points. It is one way of downscaling data, and was a motivation for Gauss, to be applied to planetary motion.

The way it works is:

# You fourier-transform your data
# You add frequecies above the Nyquist frequency (in absolute values), but set all the amplitudes of the new frequencies to zero.
# Note that the frequencies are stored such that eg. fn-1 is a low frequency -1.
# The resulting 2n array can be back transformed, and now gives an interpolated signal.

For this assignment, write an application that will read in an image from a binary file into a 2d double precision array (this will require converting from bytes to doubles), and creates an image twice the size in all directions using trigonometric interpolation. Use a real-to-half-complex version of the fftw (note: in 2d, this version of the fftw mixes fourier components with the same physical magnitude of their wave number k, so this will work).
You can process the red, green and blue values separately.

''Assignment 2''

Write an application which reads an image and performs a low pass filter on the image, i.e., any fourier components with magnitudes k larger than n/8 are to be set to zero, after which the fourier inverse is taken and the image is written out to disk again. Use the same fft technique as in the first assignment.

'''Input image'''

Use [[Media:gauss256.tgz|this image of Gauss]].

'''Image format:'''

Use the following simple PPM format:

First line (ascii): 'P6\n' 
Second line, in ascii, 'width height\n' 
Third line, in ascii, 'maxcolorvalue\n' (this is typically just 255) 
Following that, in binary, are byte-triplets with the red, green and blue values of each pixel. 
Note: in C, the 'unsigned char' data type matches the concept of a byte best (for most machines anyway).

In fact, between the first and second line, one can have comment lines that start with '#'.

=Part 3: High Performance Scientific Computing=

==Prerequisites==

Part 1 or good c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

You will need to bring a laptop with a ssh facility. Hands-on parts will be done on SciNet's GPC cluster.

For those who don't have a SciNet account yet, the instructions can be found at http://wiki.scinethpc.ca/wiki/index.php/Essentials#Accounts

==Dates==
March 19, 21, 26, and 28, 2013 
April 2, 4, 9, and 11, 2013

==Topics==
===''Lecture 1:'' Introduction to Parallel Programming ===
:::[[File:Lecture17-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.html]]
:::[[Media:Lecture17-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.mp4 Video recording]

===''Lecture 2:'' Parallel Computing Paradigms ===

:::[[File:Lecture18-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.html]]
:::[[Media:Lecture18-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.mp4 Video recording]  /   [[#HW1_3|homework 1]]

===''Lectures 3,4:'' Shared Memory Programming with OpenMP, part 1,2===

:::[[Media:Lecture19-2013.pdf|Slides]]

===''Lecture 5:'' Distributed Parallel Programming with MPI, part 1===

:::[[Media:Lecture21-2013.pdf|Slides]]

===''Lecture 6:'' Distributed Parallel Programming with MPI, part 2===

:::[[Media:Lecture22-2013.pdf|Slides]]

''Lecture 7''   Distributed Parallel Programming with MPI, part 3 
''Lecture 8''   Hybrid OpenMPI+MPI Programming

== Homework assignments ==

=== HW1 ===

* Read the SciNet tutorial (as it pertains to the GPC)
* Read the GPC Quick Start.
* Get the first set of code:
<source lang="bash">
$ cd $SCRATCH
$ git clone /scinet/course/sc3/homework1
$ cd homework1
$ source setup
$ make
</source>
*This contains threaded program 'blurppm' and 266 ppm images to be blurred. Usage:
<source lang="bash">
blurppm INPUTPPM OUTPUTPPM BLURRADIUS NUMBEROFTHREADS
</source>
* Simple test:
<source lang="bash">
$ qsub -l nodes=1:ppn=8,walltime=2:00:00 -I -X -qdebug
$ cd $SCRATCH/homework1
$ time blurppm 001.ppm new001.ppm 30 1
real 0m52.900s
user 0m52.881s
sys 0m0.008s
$ display 001.ppm &
$ display new001.ppm &
</source>
''Assignment 1''
* Time blurppm with a BLURRADIUS ranging from 1 to 41 in steps of 4, and for NUMBEROFTHREADS ranging from 1 to 16. Record the (real) duration of each run.
* Plot the duration as a function of NUMBEROFTHREADS, as well as the speed-up and efficiency.
* Submit script and plots of the duration, speedup and effiency as a function of NUMBEROFTHREADS.
''Assignment 2''
* Use GNU parallel to run blurppm on all 266 images with a radius of 41.
* Investigate different scenarios:
:# Have GNU parallel run 16 at a time with just 1 thread.
:# Have GNU parallel run 8 at a time with 2 threads.
:# Have GNU parallel run 4 at a time with 4 threads.
:# Have GNU parallel run 2 at a time with 8 threads.
:# Have GNU parallel run 1 at a time with 16 threads.
:Record the total time it takes in each of these scenarios.
* Repeat this with a BLURRADIUS of 3.
* Submit scripts, timing data and plots.

=== HW2 ===

In the course materials ( /scinet/course/ppp/nbodyc or nbodyf ) there is the source code for a serial N-body integrator. This, like the molecular dynamics you've seen earlier, calculates long-range forces by particles on all of the other particles.

OpenMP the force calculation, and present timing results for 1,4, and 8 threads compared to the serial version. Note that you can turn off graphic output by removing the "USEPGPLOT = -DPGPLOT" line in Makefile.inc in the top level directory.

Begin by doubling the work by _not_ calculating two forces at once (eg, not making use of fji = -fij), and simply parallelizing the outer force loop. Then find a way to implement the forces efficiently but also in parallel. Is there any other part of the problem which could usefully be parallelized?

=Links=

==Unix==
* Cygwin: http://www.cygwin.com
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
* Intro to unix shell from software carpentry: http://software-carpentry.org/4_0/shell

==C/C++==
* [[One-Day Scientific C++ Class]] at SciNet
* C++ library reference: http://www.cplusplus.com/reference
* C preprocessor: http://www.cprogramming.com/tutorial/cpreprocessor.html
* Boost: http://www.boost.org
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==Git==
* Git: http://git-scm.com
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
* Git cheat sheet from Git Tower: http://www.git-tower.com/files/cheatsheet/Git_Cheat_Sheet_grey.pdf

==Python==
* Python: http://www.python.org
* IPython: http://ipython.org
* Matplotlib: http://www.matplotlib.org
* Enthought python distribution: http://www.enthought.com/products/edudownload.php 
(this gives you numpy, matplotlib and ipython all installed in one fell swoop)

* Intro to python from software carpentry: http://software-carpentry.org/4_0/python
* Tutorial on matplotlib: http://conference.scipy.org/scipy2011/tutorials.php#jonathan
* Npy file format: https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==ODEs==
* Integrators for particle based ODEs (i.e. molecular dynamics): http://www.chem.utoronto.ca/~rzon/simcourse/partmd.pdf. '''Focus on 4.1.4 - 4.1.6 for practical aspects.'''
* Numerical algorithm to solve ODEs (General) in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Chapter 16

==Interpolation (2D) ==
* Interpolation in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Pages 123-128
* Wikipedia pages on [http://en.wikipedia.org/wiki/Bilinear_interpolation Bilinear Interpolation] and [http://en.wikipedia.org/wiki/Bicubic_interpolation Bicubic Interpolation] are not bad either.

==BLAS==
* [http://www.tacc.utexas.edu/tacc-projects/gotoblas2 gotoblas]
* [http://math-atlas.sourceforge.net/ ATLAS]

==LAPACK==
* http://www.netlib.org/lapack

==GSL==
* GNU Scientific Library: http://www.gnu.org/s/gsl

==FFT==
* FFTW: http://www.fftw.org

==Top500==
* TOP500 Supercomputing Sites: http://top500.org

==OpenMP==
* OpenMP (open multi-processing) application programming interface for shared memory programming: http://openmp.org

==GNU parallel==
* Official citation: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

==SciNet==

Anything on this wiki, really, but specifically:
* [[Essentials|SciNet Essentials]]
* [[GPC Quickstart]]
* [[Media:SciNet_Tutorial.pdf |SciNet User Tutorial]]
* [[Software and Libraries]]

==Other Resources==
* [http://galileo.phys.virginia.edu/classes/551.jvn.fall01/goldberg.pdf What Every Computer Scientist Should Know About Floating-Point Arithmetic] - the classic (and extremely comprehensive) overview of the basics of floating point math. The first few pages, in particular, are very useful.
* [http://arxiv.org/abs/1005.4117 Random Numbers In Scientific Computing: An Introduction] by Katzgraber. A very lucid discussion of pseudo random number generators for science.

Scientific Computing Course

2013-04-02T15:10:11Z

Ljdursi: /* Homework assignments */

''This wiki page concerns the 2013 installment of SciNet's Scientific Computing course. Material from the previous installment can be found on [[Scientific Software Development Course]], [[Numerical Tools for Physical Scientists (course)]], and [[High Performance Scientific Computing]]''

__TOC__
=Syllabus=

==About the course==
* Whole-term graduate course
* Prerequisite: basic C, C++ or Fortran experience.
* Will use `C++ light' and Python
* Topics include: Scientific computing and programming skills, Parallel programming, and Hybrid programming.

There are three parts to this course:

# Scientific Software Development: Jan/Feb 2013 ''python, C++, git, make, modular programming, debugging''
# Numerical Tools for Physical Scientists: Feb/Mar 2013 ''modelling, floating point, Monte Carlo, ODE, linear algebra,fft''
# High Performance Scientific Computing: Mar/Apr 2013 ''openmp, mpi and hybrid programming''

Each part consists of eight one-hour lectures, two per week.

These can be taken separately by astrophysics graduate students at the University of Toronto as mini-courses, and by physics graduate students at the University of Toronto as modular courses.

The first two parts count towards the SciNet Certificate in Scientific Computing, while the third part can count towards the SciNet HPC Certificate. For more info about the SciNet Certificates, see http://www.scinethpc.ca/2012/12/scinet-hpc-certificate-program.

==Location and Times==
[http://www.scinethpc.ca/2010/08/contact-us SciNet HeadQuarters] 
256 McCaul Street, Toronto, ON 
Room 229 (Conference Room) 
Tuesdays 11:00 am - 12:00 noon 
Thursdays 11:00 am - 12:00 noon

==Instructors and office hours==

* Ramses van Zon - 256 McCaul Street, Rm 228 - Mondays 3-4pm
* L. Jonathan Dursi - 256 McCaul Street, Rm 216 - Wednesdays 3-4pm

==Grading scheme==

Attendence to lectures.

Four home work sets (i.e., one per week), to be returned by email by 9:00 am the next Thursday.

==Sign up==
Sign up for this graduate course goes through SciNet's course website. The direct link is https://support.scinet.utoronto.ca/courses/?q=node/99. If you do not have a SciNet account but wish to register for this course, please email support@scinet.utoronto.ca . 
Sign up is closed.

=Part 1: Scientific Software Development=

==Prerequisites==

Some programming experience. Some unix prompt experience.

'''Software that you'll need:'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python 2, IPython, Numpy, SciPy and Matplotlib (which you all get if you use the Enthought distribution) installed on your laptop. Links are given at the bottom of this page.

==Dates==

January 15, 17, 22, 24, 29, and 31, 2013 
February 5 and 7, 2013

==Topics (with lecture slides and recordings)==

===''Lecture 1:'' C++ introduction===
:::[[File:Lecture1-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.html]]
:::[[Media:Lecture1-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.mp4 Video recording]

===''Lecture 2:'' More C++, build and version control ===
:::[[File:Lecture2-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.html]]
:::Guest lecturer: Michael Nolta (CITA) for the git portion of the lecture.
:::[[Media:Lecture2-2013.pdf|C++ and Make slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.mp4 C++ and Make video recording]  /   [[Media:Git-Nolta.pdf|Git slides]]  /   [[#HW1|Homework assigment 1]]

===''Lecture 3:'' Python and visualization===
:::[[File:Lecture3-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.html]]
:::[[Media:Lecture3-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.mp4 Video recording]

===''Lecture 4:'' Modular programming, refactoring, testing===
:::[[File:Lecture4-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.html]]
:::[[Media:Lecture4-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.mp4 Video recording]  /   [[#HW2|Homework assigment 2]]
:::[http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc (course project source file)]  /   [http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py (corresponding python movie generator)]

===''Lecture 5:'' Object oriented programming===
:::[[Media:Lecture5-2013.pdf|Slides]]
:::Recordings of this lecture are missing, but you could view the videos of SciNet's [[One-Day Scientific C++ Class]], in particular the parts on classes, polymorphism, and inheritance.

===''Lecture 6:'' ODE, interpolation===
:::[[File:Lecture6-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.html]]
:::[[Media:ScientificComputing2013-Lecture5-ODE.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.mp4 Video recording]  /   [[#HW3|Homework assigment 3]]

===''Lecture 7:'' Development tools: debugging and profiling===
:::[[File:Lecture7-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.html]]
:::[[Media:ScientificComputing2013-Debugging.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.mp4 Video recording]

===''Lecture 8:'' Objects in Python, linking C++ and Python===
:::[[File:Lecture8-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.html]]
:::[[Media:Lecture8-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.mp4 Video recording]

==Homework assignments==

===HW1===

'''''Multi-file C++ program to create a data file'''''

We’ve learned programming in basic C++, use of make and Makefiles to build projects, and local use of git for version control. In this first assignment, you’ll use these to make a multi-file C++ program, built with make, which computes and outputs a data file.

* Start a git repository, and begin writing a C++ program to
:# Get an array size and a standard deviation from user input,
:# Allocate a 2d array (use the code given in lecture 2),
:# Store a 2d Gaussian with a maximum at the centre of the array and given standard deviation (in units of grid points),
:# Output that array to a text file,
:# Free the array, and exit.
* The output text file should contain just the data in text format, with a row of the file corresponding to a row of the array and with whitespace between the numbers.
* The 2d array creation/freeing routines should be in one file (with an associated header file), the gaussian calculation be in another (ditto), and the output routine be in a third, with the main program calling each of these.
* Use a makefile to build your code (add it to the repository).
* You can start with everything in one file, with hardcoded values for sizes and standard deviation and a static array, then refactor things into multiple files, adding the other features.
* As a test, use the ipython executable that came with your Enthought python distribution to read your data and plot it. If your data file is named ‘data.txt’, running the following:
<pre>
$ ipython --pylab
In [1]: data = numpy.genfromtxt('data.txt')
In [2]: contour(data)
</pre>
should give a nice contour plot of a 2-dimensional gaussian.
* Email in your source code, makefile and the "git log" output of all your commits by email by at 9:00 am Thursday Jan 24th, 2013. Please zip or tar these files together as one attachment, with a file name that includes your name and "HW1".

===HW2===
'''''Refactor legacy code to a modular project with unit tests'''''

In class, today, we talked about modular programming and testing, and the project we’ll be working on for the next three weeks. This homework will start advancing on that project by working on the “legacy” code given to us by our supervisor ([http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc]), with a corresponding python plotting script ([http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py]), and whipping it into shape before we start adding new physics.
* Start a git repository for this project, and add the two files.
* Create a Makefile and add it to the repository.
* Since we have no tests, run the program with console output redirected to a file:
:<pre>$ diffuse > original-output.txt</pre>
 ''It turns out the code has a bug that can make the output different when the same code is run again, which obviously would not be good for a baseline test. Replace 'float error;' by 'float error=0.0;' to fix this.''
* Also save the two .npy output files, e.g. to original-data.npy and original-theory.npy. The triplet of files (original-output.txt, original-data.npy and original-theory.npy) serve as a baseline integrated test (add these to repository).
* Then write a 'test' target in your makefile that:
** Runs 'diffuse' with output to a new file.
** Compares the file with the baseline test file, and compare the .npy files.
:: (hint: the unix command diff or cmp can compare files).
* First refactoring: Move the global variables into the main routine.
* ''Chorus: Test your modified code, and commit.''
* Second refactoring: Extract a diffusion operator routine, that gets called from main.
* ''Chorus''
* Create a .cc/.h module for the diffusion operator.
* ''Chorus''
* Add two tests for the diffusion operator: for a constant and for a linear input field (<tt>rho[i][j]=a*i+b*j</tt>). Add these to the test target in the makefile.
* ''Chorus''
* More refactoring: Extract three more .cc/.h modules:
** for output (should not contain hardcoded filenames)
** computation of the theory
** and for the array allocation stuff.
* ''Chorus''
* Describe, but don't implement in the .h and .cc, what would be appropriate unit tests for these three modules.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by 9:00 am on Thursday January 31, 2013.

===HW3===
This week, we learned about object oriented programming, which fits nicely within the modular programming idea. In this homework, we are going to use some of it to restructure our code and get it ready to add the tracer particle, the goal of the course project.

The goal will be to have an instance of a <tt>Diffusion</tt> class,
as well as an instance of <tt>Tracer</tt>, which for now will be a
free particle moving as ('''x'''(t),'''y'''(t)) = ('''x'''(0) +
'''vx''' t, '''y'''(0) + '''vy''' t), without any coupling yet (we
will handle this next week).

To be more specific:
* Clean up your code, using the feedback from your HW2 grading, such that the modules are as independent as possible.
* If you have not done so yet, add comments to the header files of your modules to explain exactly what each function does (without going into implementation details), what its arguments mean and what it returns (unless it's a void function, of course).
* Objectify the <tt>main</tt> routine, by creating a class <tt>Diffusion</tt>.
* Put this class in its own module (declaration in .h, implementation in .cc). For instance, the declaration could be
<pre>
// diffusion.h
#ifndef DIFFUSIONH
#define DIFFUSIONH
#include <fstream>
class Diffusion {
public:
Diffusion(float x1, float x2, float D, int numPoints);
void init(float a0, float sigma0); // set initial field
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
float getRho(int i, int j); // get a value of the field
~Diffusion();
private:
float*** rho;
...
};
#endif
</pre>
(this is not supposed to be prescriptive.)
* In the implementation file you'd have things like
<pre>
// diffusion.cc
#include "diffusion.h"
...
void Diffusion::timeStep(float dt)
{
// code for the timeStep ...
}
...
</pre>
(note the inclusion of the module's header file on the top of the implementation, so the class is declared).
* Let <tt>int main()</tt> have the same functionality as before, but now by defining the parameters of the run, creating an object of this class, setting up file streams, and taking time steps and writing out by using calls to member functions of this object.
* Additionally, write a class <tt>Tracer</tt> which for now implements a free particle in 2d. Something like:
<pre>
class Tracer {
public:
Tracer(float x1, float x2);
void init(float x0, float y0, float vx, float vy);
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
~Tracer();
private:
...
};
</pre>
:The timeStep implementation can in this case use the infamous forward Euler integration scheme, because it happens to be exact here.
:When it comes to output to a npy file, let's view the the data of the tracer particle at one point in time as a 2x2 matrix <tt>[[x,y],[vx,vy]]</tt>, so we can use much of the npy output code that we used for the diffusion field, which was a (numPoints+2)x(numPoints+2) matrix.
* This class too should be its own module (Often, "one class, one module" is a good paradigm, though occasionally you'll have closely related classes).
* Add some code to int main to have the Tracer particle evolve at the same time as the diffusion field (although the two are completely uncoupled).
* Keep using git and make, run the tests that you have regularly to make sure your program still works.

Note that because we've now set up our program in a modular fashion, you can do
different parts of this assignment in any order you want. For instance, to wrap your head around object oriented programming, you may like implementing the tracer particle first, so that your diffusion code stays intact. Or you might want to wait with commenting until the end if you think you'll have to change a module for this assignment.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by
3:00 pm on Friday February 8, 2013.

===HW4===

In this homework, we are going to implement the class project of a tracer particle coupled to a diffusion equation.
The full specification of the physical problem is [[Media:ScClassProject.pdf|here]].
* Augment the tracer particle to include a force in the x and in the y direction, and a friction coefficient alpha, which at first can be constant.
* Implement the so-called leapfrog integration algorithm for the tracer particle
:::v ← v + f(v) Δt / m
:::r ← r + v Δt
:where v,r, and f are 2d vectors and f(v) is the total, velocity dependence force speficied in the class project, i.e., the sum of the external force F=qE and the friction force -αv. (Note: the v dependence of f make this strictly not a leapfrog integration, but we'll ignore that here.)
* Further augment the tracer class with a member function 'couple' which takes a diffusion field as input, and adjusts the friction constant.
* Your implementation of the 'couple' member function will need to interpolate the diffusion field to the current position of the particle. Use [[Media:CppInterpolation.tgz|this interpolation module]].
* Rewrite your main routine so that before tracer's time step, one calls the coupling. You may need to modify the Diffusion class a bit to get <tt>rho[active]</tt> out.
* For simplicity, use the same time step for both the diffusion and the tracer particle.
* Keep using git and make.

You will hand in your source code, makefiles and the git log file of all your commits by email by 9:00 am on Thursday February 21, 2013. Email the files, preferably zipped or tarred, to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

=Part 2: Numerical Tools for Physical Scientists=

==Prerequisites==

Part 1 or solid c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python (Enthought) installed on your laptop.

==Dates==

February 12, 14, 26, and 28, 2013 
March 5, 7, 12, and 14, 2013

==Topics==

===''Lecture 1:'' Numerics ===
:::[[File:Lecture9-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.html]]
:::[[Media:Lecture9-2013-Numerics.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.mp4 Video recording]

===''Lecture 2:'' Random numbers ===
:::[[File:Lecture10-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.html]]
:::[[Media:Lecture10-2013-PRNG.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW1_2 Homework assignment 1]

===''Lecture 3:'' Numerical integration and ODEs ===
:::[[File:Lecture11-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.html]]
:::[[Media:Lecture11-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.mp4 Video recording]

===''Lecture 4:'' Molecular Dynamics ===
:::[[File:Lecture12-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.html]]
:::[[Media:Lecture12-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW2_2 Homework assignment 2]

===''Lecture 5:'' Linear Algebra part I ===
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 6)]]

===''Lecture 6:'' Linear Algebra part II and PDEs===
:::[[File:Lecture14-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.html]]
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 5)]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW3_2 Homework assignment 3]

===''Lecture 7:'' Fast Fourier Transform===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.html]]
:::[[Media:Lecture15-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.mp4 Video recording]  /  [[Media:Sincfftw.cc|example code]]

===''Lecture 8:'' FFT for real and multidimensional data===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.html]]
:::[[Media:Lecture16-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.mp4 Video recording]  /   [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW4_2 Homework assignment 4]

==Homework Assignments==

===HW1===
This week's homework consists of two assignments.

''Assignment 1''

* Consider the sequence of numbers: 1 followed by 108 values of 10-8
* Should sum to 2
* Write code which sums up those values in order. What answer does it get?
* Add to program routine which sums up values in reverse order. Does it get correct answer?
* How would you get correct answer?
* Submit code, Makefile, text file with answers.

''Assignment 2''

* Implement an linear congruential generator with a = 106, c = 1283, m = 6075 that generates random numbers from 0..1
* Using that and MT: generate 10,000 pairs (dx, dy) with dx, dy each in -0.1 .. +0.1. Generate histograms of dx and dy (say 200 bins). Does it look okay? What would you expect variation to be?
* For 10,000 points: take random walks from (x,y)=(0,0) until exceed radius of 2, then stop. Plot histogram of final angles for the two psuedo random number generators. What do you see?
* Submit makefile, code, plots, git log.

Both assignments due on Thursday Feb 28th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW2===

''Assignment 1''

* Compute numerically (using the GSL):

::∫03 f(x)  dx

:(that is the integral of f(x) from x=0 to x=3)

:with

::f(x) = ln(x) sin(x) e-x

:using three different methods:
# Extended Simpsons' rule
# Gauss-Legendre quadrature
# Monte Carlo sampling

*Hint: what is f(0)?

*Compare the convergence of these methods by increasing number of function evaluations.

*Submit makefile, code, plots, version control log.

''Assignment 2''

* Using an adaptive 4th order Runge-Kutta approach, with a relative accuracy of 1e-4, compute the solution for t = [0,100] of the following set of coupled ODEs (Lorenz oscillator)

::dx/dt = σ(y - x)

::dy/dt = (ρ-z)x-y

::dz/dt = xy - βz

:with σ=10; β=8/3; ρ = 28, and with initial conditions

::x(0) = 10

::y(0) = 20

::z(0) = 30

* Hint: study the GSL documentation.

*Submit makefile, code, plots, version control log.

Both assignments due on Thursday Mar 7th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW3===

Part 1:

The time-explicit formulation of the 1d diffusion equation looks like this:
<math>
\begin{eqnarray*}
q^{n+1} & = & q^n + \frac{D \Delta t}{\Delta x^2}
\left (
\begin{matrix}
-2 & 1 \\
1 & -2 & 1 \\
& 1 & -2 & 1 \\
& & & \cdots & \\
& & & 1 & -2 & 1 \\
& & & & 1 & -2 \\
\end{matrix}
\right ) q^n \\
& = & \left ( 1 + \frac{D \Delta t}{\Delta x^2} A \right ) q^n
\end{eqnarray*}
</math>

what are the eignvalues of the matrix A? What modes would we expect to be amplified/damped by this operator?

* Consider 100 points in the discretization (eg, A is 100x100)
* Calculate the eigenvalues and eigenvectors (using D__EV ; which sort of matrix are we using here?)
* Plot the modes with the largest and smallest absolute-value of eigenvalues, and explain their physical significance
* The numerical method will become unstable with one eigenmode $v$ begins to grow uncontrollably whenever it is present, e.g.
$ \frac{D \Delta t}{\Delta x^2} A v = \frac{D \Delta t}{\Delta x^2} \lambda v > v$. In a timestepping solution, the only way to avoid this for a given physical set of parameters and grid size is to reduce the timestep, $\Delta t$. Use the largest absolute value eigenvalue to place a constraint on $\Delta t$ for stability.

Part 2:

Using the above constraint on $\Delta t$, for a 1d grid of size 100 (eg, a 100x100 matrix A), using lapack, evolve this PDE. Plot and explain results.

* Have an initial condition of $q(x=0,t=0) = 1$, and $q(t=0)$ everywhere else being zero (eg, hot plate just turned on at the left)
* Take ~100 timesteps and plot the the evolution of $q(x,t)$ at 5 times over that period.
* You’ll want to use a matrix multiply to compute the matrix-vector multiply ( http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html). Do multiply in double precision (D__MV). Which should you use?
* The GSL has a cblas interface, http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html ; an example of its use can be found here http://www.gnu.org/software/gsl/manual/html_node/GSL-CBLAS-Examples.html

Important things to know about lapack:
* If you are using an nxn array, the “leading dimension” of the array is n. (This argument is so that you could work on sub-matrices if you wanted)
* Have to make sure the 2d array is contiguous block of memory
* You'll (presumably) want to use the C bindings for LAPACK - [http://www.netlib.org/lapack/lapacke.html lapacke]. Note that the usual C arrays are row-major.

Here's a simple example of calling a LAPACKE routine; note that how the matrix is described (here with a pointer to the data, a leading dimension, and the number of rows and columns) will vary with different types of matrix:

<source lang="cpp">
#include <iostream>
#include <mkl_lapacke.h>

double **matrix(int n,int m);
void free_matrix(double **a);

int main (int argc, const char * argv[])
{

const int n=5; // number of rows, columns of the matrix
const int m = n; // nrows
const int leading_dim_A=n; // leading dimension (# of cols for row major);
// lets us operate on sub-matrices in principle
const int leading_dim_b=n; // similarly for b
double **A;
double *b;

b = new double[leading_dim_b];
A = matrix(n,leading_dim_A);

for (int i=0; i<n; i++)
for (int j=0; j<leading_dim_A; j++)
A[i][j] = 0.;

// let's do a trivial solve
// It should be pretty clear that the solution to this system
// is x = {0,1,2...n-1}

for (int i=0; i<leading_dim_A; i++) {
A[i][i] = 2.;
}

for (int i=0; i<leading_dim_b; i++) {
b[i] = 2*i;
}

const char transpose='N'; //solve Ax=b, not A^T x = b
const int nrhs = 1; // we're only solving 1 right hand side
int info;

// Call DGELS; b will be overwritten with the value of x.
info = LAPACKE_dgels(LAPACK_COL_MAJOR,transpose,m,n,nrhs,
&(A[0][0]),leading_dim_A, &(b[0]),leading_dim_b);

// print results
for(int i=0;i<n;i++)
{
if (i != n/2)
std::cout << " " << b[i] << std::endl;
else
std::cout << "x = " << b[i] << std::endl;
}
return(info);
}

double **matrix(int n,int m) {
double **a = new double * [n];
a[0] = new double [n*m];

for (int i=1; i<n; i++)
a[i] = &a[0][i*m];

return a;
}

void free_matrix(double **a) {
delete[] a[0];
delete[] a;
}
</source>

===HW4===

''Assignment 1''

Trigonometric interpolation uses a n point Fourier series to find values at intermediate points. It is one way of downscaling data, and was a motivation for Gauss, to be applied to planetary motion.

The way it works is:

# You fourier-transform your data
# You add frequecies above the Nyquist frequency (in absolute values), but set all the amplitudes of the new frequencies to zero.
# Note that the frequencies are stored such that eg. fn-1 is a low frequency -1.
# The resulting 2n array can be back transformed, and now gives an interpolated signal.

For this assignment, write an application that will read in an image from a binary file into a 2d double precision array (this will require converting from bytes to doubles), and creates an image twice the size in all directions using trigonometric interpolation. Use a real-to-half-complex version of the fftw (note: in 2d, this version of the fftw mixes fourier components with the same physical magnitude of their wave number k, so this will work).
You can process the red, green and blue values separately.

''Assignment 2''

Write an application which reads an image and performs a low pass filter on the image, i.e., any fourier components with magnitudes k larger than n/8 are to be set to zero, after which the fourier inverse is taken and the image is written out to disk again. Use the same fft technique as in the first assignment.

'''Input image'''

Use [[Media:gauss256.tgz|this image of Gauss]].

'''Image format:'''

Use the following simple PPM format:

First line (ascii): 'P6\n' 
Second line, in ascii, 'width height\n' 
Third line, in ascii, 'maxcolorvalue\n' (this is typically just 255) 
Following that, in binary, are byte-triplets with the red, green and blue values of each pixel. 
Note: in C, the 'unsigned char' data type matches the concept of a byte best (for most machines anyway).

In fact, between the first and second line, one can have comment lines that start with '#'.

=Part 3: High Performance Scientific Computing=

==Prerequisites==

Part 1 or good c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

You will need to bring a laptop with a ssh facility. Hands-on parts will be done on SciNet's GPC cluster.

For those who don't have a SciNet account yet, the instructions can be found at http://wiki.scinethpc.ca/wiki/index.php/Essentials#Accounts

==Dates==
March 19, 21, 26, and 28, 2013 
April 2, 4, 9, and 11, 2013

==Topics==
===''Lecture 1:'' Introduction to Parallel Programming ===
:::[[File:Lecture17-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.html]]
:::[[Media:Lecture17-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.mp4 Video recording]

===''Lecture 2:'' Parallel Computing Paradigms ===

:::[[File:Lecture18-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.html]]
:::[[Media:Lecture18-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.mp4 Video recording]  /   [[#HW1_3|homework 1]]

===''Lectures 3,4:'' Shared Memory Programming with OpenMP, part 1,2===

:::[[Media:Lecture19-2013.pdf|Slides]]

===''Lecture 5:'' Distributed Parallel Programming with MPI, part 1===

:::[[Media:Lecture21-2013.pdf|Slides]]

''Lecture 6''   Distributed Parallel Programming with MPI, part 2 
''Lecture 7''   Distributed Parallel Programming with MPI, part 3 
''Lecture 8''   Hybrid OpenMPI+MPI Programming

== Homework assignments ==

=== HW1 ===

* Read the SciNet tutorial (as it pertains to the GPC)
* Read the GPC Quick Start.
* Get the first set of code:
<source lang="bash">
$ cd $SCRATCH
$ git clone /scinet/course/sc3/homework1
$ cd homework1
$ source setup
$ make
</source>
*This contains threaded program 'blurppm' and 266 ppm images to be blurred. Usage:
<source lang="bash">
blurppm INPUTPPM OUTPUTPPM BLURRADIUS NUMBEROFTHREADS
</source>
* Simple test:
<source lang="bash">
$ qsub -l nodes=1:ppn=8,walltime=2:00:00 -I -X -qdebug
$ cd $SCRATCH/homework1
$ time blurppm 001.ppm new001.ppm 30 1
real 0m52.900s
user 0m52.881s
sys 0m0.008s
$ display 001.ppm &
$ display new001.ppm &
</source>
''Assignment 1''
* Time blurppm with a BLURRADIUS ranging from 1 to 41 in steps of 4, and for NUMBEROFTHREADS ranging from 1 to 16. Record the (real) duration of each run.
* Plot the duration as a function of NUMBEROFTHREADS, as well as the speed-up and efficiency.
* Submit script and plots of the duration, speedup and effiency as a function of NUMBEROFTHREADS.
''Assignment 2''
* Use GNU parallel to run blurppm on all 266 images with a radius of 41.
* Investigate different scenarios:
:# Have GNU parallel run 16 at a time with just 1 thread.
:# Have GNU parallel run 8 at a time with 2 threads.
:# Have GNU parallel run 4 at a time with 4 threads.
:# Have GNU parallel run 2 at a time with 8 threads.
:# Have GNU parallel run 1 at a time with 16 threads.
:Record the total time it takes in each of these scenarios.
* Repeat this with a BLURRADIUS of 3.
* Submit scripts, timing data and plots.

=== HW2 ===

In the course materials ( /scinet/course/ppp/nbodyc or nbodyf ) there is the source code for a serial N-body integrator. This, like the molecular dynamics you've seen earlier, calculates long-range forces by particles on all of the other particles.

OpenMP the force calculation, and present timing results for 1,4, and 8 threads compared to the serial version. Note that you can turn off graphic output by removing the "USEPGPLOT = -DPGPLOT" line in Makefile.inc in the top level directory.

Begin by doubling the work by _not_ calculating two forces at once (eg, not making use of fji = -fij), and simply parallelizing the outer force loop. Then find a way to implement the forces efficiently but also in parallel. Is there any other part of the problem which could usefully be parallelized?

=Links=

==Unix==
* Cygwin: http://www.cygwin.com
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
* Intro to unix shell from software carpentry: http://software-carpentry.org/4_0/shell

==C/C++==
* [[One-Day Scientific C++ Class]] at SciNet
* C++ library reference: http://www.cplusplus.com/reference
* C preprocessor: http://www.cprogramming.com/tutorial/cpreprocessor.html
* Boost: http://www.boost.org
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==Git==
* Git: http://git-scm.com
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
* Git cheat sheet from Git Tower: http://www.git-tower.com/files/cheatsheet/Git_Cheat_Sheet_grey.pdf

==Python==
* Python: http://www.python.org
* IPython: http://ipython.org
* Matplotlib: http://www.matplotlib.org
* Enthought python distribution: http://www.enthought.com/products/edudownload.php 
(this gives you numpy, matplotlib and ipython all installed in one fell swoop)

* Intro to python from software carpentry: http://software-carpentry.org/4_0/python
* Tutorial on matplotlib: http://conference.scipy.org/scipy2011/tutorials.php#jonathan
* Npy file format: https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==ODEs==
* Integrators for particle based ODEs (i.e. molecular dynamics): http://www.chem.utoronto.ca/~rzon/simcourse/partmd.pdf. '''Focus on 4.1.4 - 4.1.6 for practical aspects.'''
* Numerical algorithm to solve ODEs (General) in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Chapter 16

==Interpolation (2D) ==
* Interpolation in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Pages 123-128
* Wikipedia pages on [http://en.wikipedia.org/wiki/Bilinear_interpolation Bilinear Interpolation] and [http://en.wikipedia.org/wiki/Bicubic_interpolation Bicubic Interpolation] are not bad either.

==BLAS==
* [http://www.tacc.utexas.edu/tacc-projects/gotoblas2 gotoblas]
* [http://math-atlas.sourceforge.net/ ATLAS]

==LAPACK==
* http://www.netlib.org/lapack

==GSL==
* GNU Scientific Library: http://www.gnu.org/s/gsl

==FFT==
* FFTW: http://www.fftw.org

==Top500==
* TOP500 Supercomputing Sites: http://top500.org

==OpenMP==
* OpenMP (open multi-processing) application programming interface for shared memory programming: http://openmp.org

==GNU parallel==
* Official citation: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

==SciNet==

Anything on this wiki, really, but specifically:
* [[Essentials|SciNet Essentials]]
* [[GPC Quickstart]]
* [[Media:SciNet_Tutorial.pdf |SciNet User Tutorial]]
* [[Software and Libraries]]

==Other Resources==
* [http://galileo.phys.virginia.edu/classes/551.jvn.fall01/goldberg.pdf What Every Computer Scientist Should Know About Floating-Point Arithmetic] - the classic (and extremely comprehensive) overview of the basics of floating point math. The first few pages, in particular, are very useful.
* [http://arxiv.org/abs/1005.4117 Random Numbers In Scientific Computing: An Introduction] by Katzgraber. A very lucid discussion of pseudo random number generators for science.

File:Lecture21-2013.pdf

2013-04-02T13:59:17Z

Ljdursi:

Scientific Computing Course

2013-04-02T13:58:49Z

Ljdursi: /* Topics */

''This wiki page concerns the 2013 installment of SciNet's Scientific Computing course. Material from the previous installment can be found on [[Scientific Software Development Course]], [[Numerical Tools for Physical Scientists (course)]], and [[High Performance Scientific Computing]]''

__TOC__
=Syllabus=

==About the course==
* Whole-term graduate course
* Prerequisite: basic C, C++ or Fortran experience.
* Will use `C++ light' and Python
* Topics include: Scientific computing and programming skills, Parallel programming, and Hybrid programming.

There are three parts to this course:

# Scientific Software Development: Jan/Feb 2013 ''python, C++, git, make, modular programming, debugging''
# Numerical Tools for Physical Scientists: Feb/Mar 2013 ''modelling, floating point, Monte Carlo, ODE, linear algebra,fft''
# High Performance Scientific Computing: Mar/Apr 2013 ''openmp, mpi and hybrid programming''

Each part consists of eight one-hour lectures, two per week.

These can be taken separately by astrophysics graduate students at the University of Toronto as mini-courses, and by physics graduate students at the University of Toronto as modular courses.

The first two parts count towards the SciNet Certificate in Scientific Computing, while the third part can count towards the SciNet HPC Certificate. For more info about the SciNet Certificates, see http://www.scinethpc.ca/2012/12/scinet-hpc-certificate-program.

==Location and Times==
[http://www.scinethpc.ca/2010/08/contact-us SciNet HeadQuarters] 
256 McCaul Street, Toronto, ON 
Room 229 (Conference Room) 
Tuesdays 11:00 am - 12:00 noon 
Thursdays 11:00 am - 12:00 noon

==Instructors and office hours==

* Ramses van Zon - 256 McCaul Street, Rm 228 - Mondays 3-4pm
* L. Jonathan Dursi - 256 McCaul Street, Rm 216 - Wednesdays 3-4pm

==Grading scheme==

Attendence to lectures.

Four home work sets (i.e., one per week), to be returned by email by 9:00 am the next Thursday.

==Sign up==
Sign up for this graduate course goes through SciNet's course website. The direct link is https://support.scinet.utoronto.ca/courses/?q=node/99. If you do not have a SciNet account but wish to register for this course, please email support@scinet.utoronto.ca . 
Sign up is closed.

=Part 1: Scientific Software Development=

==Prerequisites==

Some programming experience. Some unix prompt experience.

'''Software that you'll need:'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python 2, IPython, Numpy, SciPy and Matplotlib (which you all get if you use the Enthought distribution) installed on your laptop. Links are given at the bottom of this page.

==Dates==

January 15, 17, 22, 24, 29, and 31, 2013 
February 5 and 7, 2013

==Topics (with lecture slides and recordings)==

===''Lecture 1:'' C++ introduction===
:::[[File:Lecture1-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.html]]
:::[[Media:Lecture1-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture1-2013/lecture1-2013.mp4 Video recording]

===''Lecture 2:'' More C++, build and version control ===
:::[[File:Lecture2-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.html]]
:::Guest lecturer: Michael Nolta (CITA) for the git portion of the lecture.
:::[[Media:Lecture2-2013.pdf|C++ and Make slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture2-2013/lecture2-2013.mp4 C++ and Make video recording]  /   [[Media:Git-Nolta.pdf|Git slides]]  /   [[#HW1|Homework assigment 1]]

===''Lecture 3:'' Python and visualization===
:::[[File:Lecture3-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.html]]
:::[[Media:Lecture3-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture3-2013/lecture3-2013.mp4 Video recording]

===''Lecture 4:'' Modular programming, refactoring, testing===
:::[[File:Lecture4-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.html]]
:::[[Media:Lecture4-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture4-2013/lecture4-2013.mp4 Video recording]  /   [[#HW2|Homework assigment 2]]
:::[http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc (course project source file)]  /   [http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py (corresponding python movie generator)]

===''Lecture 5:'' Object oriented programming===
:::[[Media:Lecture5-2013.pdf|Slides]]
:::Recordings of this lecture are missing, but you could view the videos of SciNet's [[One-Day Scientific C++ Class]], in particular the parts on classes, polymorphism, and inheritance.

===''Lecture 6:'' ODE, interpolation===
:::[[File:Lecture6-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.html]]
:::[[Media:ScientificComputing2013-Lecture5-ODE.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture6-2013/lecture6-2013.mp4 Video recording]  /   [[#HW3|Homework assigment 3]]

===''Lecture 7:'' Development tools: debugging and profiling===
:::[[File:Lecture7-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.html]]
:::[[Media:ScientificComputing2013-Debugging.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture7-2013/lecture7-2013.mp4 Video recording]

===''Lecture 8:'' Objects in Python, linking C++ and Python===
:::[[File:Lecture8-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.html]]
:::[[Media:Lecture8-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture8-2013/lecture8-2013.mp4 Video recording]

==Homework assignments==

===HW1===

'''''Multi-file C++ program to create a data file'''''

We’ve learned programming in basic C++, use of make and Makefiles to build projects, and local use of git for version control. In this first assignment, you’ll use these to make a multi-file C++ program, built with make, which computes and outputs a data file.

* Start a git repository, and begin writing a C++ program to
:# Get an array size and a standard deviation from user input,
:# Allocate a 2d array (use the code given in lecture 2),
:# Store a 2d Gaussian with a maximum at the centre of the array and given standard deviation (in units of grid points),
:# Output that array to a text file,
:# Free the array, and exit.
* The output text file should contain just the data in text format, with a row of the file corresponding to a row of the array and with whitespace between the numbers.
* The 2d array creation/freeing routines should be in one file (with an associated header file), the gaussian calculation be in another (ditto), and the output routine be in a third, with the main program calling each of these.
* Use a makefile to build your code (add it to the repository).
* You can start with everything in one file, with hardcoded values for sizes and standard deviation and a static array, then refactor things into multiple files, adding the other features.
* As a test, use the ipython executable that came with your Enthought python distribution to read your data and plot it. If your data file is named ‘data.txt’, running the following:
<pre>
$ ipython --pylab
In [1]: data = numpy.genfromtxt('data.txt')
In [2]: contour(data)
</pre>
should give a nice contour plot of a 2-dimensional gaussian.
* Email in your source code, makefile and the "git log" output of all your commits by email by at 9:00 am Thursday Jan 24th, 2013. Please zip or tar these files together as one attachment, with a file name that includes your name and "HW1".

===HW2===
'''''Refactor legacy code to a modular project with unit tests'''''

In class, today, we talked about modular programming and testing, and the project we’ll be working on for the next three weeks. This homework will start advancing on that project by working on the “legacy” code given to us by our supervisor ([http://wiki.scinethpc.ca/wiki/images/f/f0/diffuse.cc diffuse.cc]), with a corresponding python plotting script ([http://wiki.scinethpc.ca/wiki/images/f/f0/plotdata.py plotdata.py]), and whipping it into shape before we start adding new physics.
* Start a git repository for this project, and add the two files.
* Create a Makefile and add it to the repository.
* Since we have no tests, run the program with console output redirected to a file:
:<pre>$ diffuse > original-output.txt</pre>
 ''It turns out the code has a bug that can make the output different when the same code is run again, which obviously would not be good for a baseline test. Replace 'float error;' by 'float error=0.0;' to fix this.''
* Also save the two .npy output files, e.g. to original-data.npy and original-theory.npy. The triplet of files (original-output.txt, original-data.npy and original-theory.npy) serve as a baseline integrated test (add these to repository).
* Then write a 'test' target in your makefile that:
** Runs 'diffuse' with output to a new file.
** Compares the file with the baseline test file, and compare the .npy files.
:: (hint: the unix command diff or cmp can compare files).
* First refactoring: Move the global variables into the main routine.
* ''Chorus: Test your modified code, and commit.''
* Second refactoring: Extract a diffusion operator routine, that gets called from main.
* ''Chorus''
* Create a .cc/.h module for the diffusion operator.
* ''Chorus''
* Add two tests for the diffusion operator: for a constant and for a linear input field (<tt>rho[i][j]=a*i+b*j</tt>). Add these to the test target in the makefile.
* ''Chorus''
* More refactoring: Extract three more .cc/.h modules:
** for output (should not contain hardcoded filenames)
** computation of the theory
** and for the array allocation stuff.
* ''Chorus''
* Describe, but don't implement in the .h and .cc, what would be appropriate unit tests for these three modules.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by 9:00 am on Thursday January 31, 2013.

===HW3===
This week, we learned about object oriented programming, which fits nicely within the modular programming idea. In this homework, we are going to use some of it to restructure our code and get it ready to add the tracer particle, the goal of the course project.

The goal will be to have an instance of a <tt>Diffusion</tt> class,
as well as an instance of <tt>Tracer</tt>, which for now will be a
free particle moving as ('''x'''(t),'''y'''(t)) = ('''x'''(0) +
'''vx''' t, '''y'''(0) + '''vy''' t), without any coupling yet (we
will handle this next week).

To be more specific:
* Clean up your code, using the feedback from your HW2 grading, such that the modules are as independent as possible.
* If you have not done so yet, add comments to the header files of your modules to explain exactly what each function does (without going into implementation details), what its arguments mean and what it returns (unless it's a void function, of course).
* Objectify the <tt>main</tt> routine, by creating a class <tt>Diffusion</tt>.
* Put this class in its own module (declaration in .h, implementation in .cc). For instance, the declaration could be
<pre>
// diffusion.h
#ifndef DIFFUSIONH
#define DIFFUSIONH
#include <fstream>
class Diffusion {
public:
Diffusion(float x1, float x2, float D, int numPoints);
void init(float a0, float sigma0); // set initial field
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
float getRho(int i, int j); // get a value of the field
~Diffusion();
private:
float*** rho;
...
};
#endif
</pre>
(this is not supposed to be prescriptive.)
* In the implementation file you'd have things like
<pre>
// diffusion.cc
#include "diffusion.h"
...
void Diffusion::timeStep(float dt)
{
// code for the timeStep ...
}
...
</pre>
(note the inclusion of the module's header file on the top of the implementation, so the class is declared).
* Let <tt>int main()</tt> have the same functionality as before, but now by defining the parameters of the run, creating an object of this class, setting up file streams, and taking time steps and writing out by using calls to member functions of this object.
* Additionally, write a class <tt>Tracer</tt> which for now implements a free particle in 2d. Something like:
<pre>
class Tracer {
public:
Tracer(float x1, float x2);
void init(float x0, float y0, float vx, float vy);
void timeStep(float dt); // solve diff. equation over dt
void toFile(std::ofstream& f); // write to file (binary,no npyheader)
void toScreen(); // report a line to screen
~Tracer();
private:
...
};
</pre>
:The timeStep implementation can in this case use the infamous forward Euler integration scheme, because it happens to be exact here.
:When it comes to output to a npy file, let's view the the data of the tracer particle at one point in time as a 2x2 matrix <tt>[[x,y],[vx,vy]]</tt>, so we can use much of the npy output code that we used for the diffusion field, which was a (numPoints+2)x(numPoints+2) matrix.
* This class too should be its own module (Often, "one class, one module" is a good paradigm, though occasionally you'll have closely related classes).
* Add some code to int main to have the Tracer particle evolve at the same time as the diffusion field (although the two are completely uncoupled).
* Keep using git and make, run the tests that you have regularly to make sure your program still works.

Note that because we've now set up our program in a modular fashion, you can do
different parts of this assignment in any order you want. For instance, to wrap your head around object oriented programming, you may like implementing the tracer particle first, so that your diffusion code stays intact. Or you might want to wait with commenting until the end if you think you'll have to change a module for this assignment.

Email in your source code and the git log file of all your commits as a .zip or .tar file by email to rzon@scinethpc.ca and ljdursi@scinethpc.ca by
3:00 pm on Friday February 8, 2013.

===HW4===

In this homework, we are going to implement the class project of a tracer particle coupled to a diffusion equation.
The full specification of the physical problem is [[Media:ScClassProject.pdf|here]].
* Augment the tracer particle to include a force in the x and in the y direction, and a friction coefficient alpha, which at first can be constant.
* Implement the so-called leapfrog integration algorithm for the tracer particle
:::v ← v + f(v) Δt / m
:::r ← r + v Δt
:where v,r, and f are 2d vectors and f(v) is the total, velocity dependence force speficied in the class project, i.e., the sum of the external force F=qE and the friction force -αv. (Note: the v dependence of f make this strictly not a leapfrog integration, but we'll ignore that here.)
* Further augment the tracer class with a member function 'couple' which takes a diffusion field as input, and adjusts the friction constant.
* Your implementation of the 'couple' member function will need to interpolate the diffusion field to the current position of the particle. Use [[Media:CppInterpolation.tgz|this interpolation module]].
* Rewrite your main routine so that before tracer's time step, one calls the coupling. You may need to modify the Diffusion class a bit to get <tt>rho[active]</tt> out.
* For simplicity, use the same time step for both the diffusion and the tracer particle.
* Keep using git and make.

You will hand in your source code, makefiles and the git log file of all your commits by email by 9:00 am on Thursday February 21, 2013. Email the files, preferably zipped or tarred, to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

=Part 2: Numerical Tools for Physical Scientists=

==Prerequisites==

Part 1 or solid c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

A unix-like environment with the GNU compiler suite (e.g. Cygwin), and Python (Enthought) installed on your laptop.

==Dates==

February 12, 14, 26, and 28, 2013 
March 5, 7, 12, and 14, 2013

==Topics==

===''Lecture 1:'' Numerics ===
:::[[File:Lecture9-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.html]]
:::[[Media:Lecture9-2013-Numerics.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture9-2013/lecture9-2013.mp4 Video recording]

===''Lecture 2:'' Random numbers ===
:::[[File:Lecture10-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.html]]
:::[[Media:Lecture10-2013-PRNG.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture10-2013/lecture10-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW1_2 Homework assignment 1]

===''Lecture 3:'' Numerical integration and ODEs ===
:::[[File:Lecture11-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.html]]
:::[[Media:Lecture11-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture11-2013/lecture11-2013.mp4 Video recording]

===''Lecture 4:'' Molecular Dynamics ===
:::[[File:Lecture12-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.html]]
:::[[Media:Lecture12-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture12-2013/lecture12-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW2_2 Homework assignment 2]

===''Lecture 5:'' Linear Algebra part I ===
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 6)]]

===''Lecture 6:'' Linear Algebra part II and PDEs===
:::[[File:Lecture14-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.html]]
:::[[Media:Lecture13-2013.pdf|Slides (combined with lecture 5)]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture14-2013/lecture14-2013.mp4 Video recording]  /  [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW3_2 Homework assignment 3]

===''Lecture 7:'' Fast Fourier Transform===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.html]]
:::[[Media:Lecture15-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture15-2013/lecture15-2013.mp4 Video recording]  /  [[Media:Sincfftw.cc|example code]]

===''Lecture 8:'' FFT for real and multidimensional data===
:::[[File:Lecture15-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.html]]
:::[[Media:Lecture16-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture16-2013/lecture16-2013.mp4 Video recording]  /   [http://wiki.scinethpc.ca/wiki/index.php/Scientific_Computing_Course#HW4_2 Homework assignment 4]

==Homework Assignments==

===HW1===
This week's homework consists of two assignments.

''Assignment 1''

* Consider the sequence of numbers: 1 followed by 108 values of 10-8
* Should sum to 2
* Write code which sums up those values in order. What answer does it get?
* Add to program routine which sums up values in reverse order. Does it get correct answer?
* How would you get correct answer?
* Submit code, Makefile, text file with answers.

''Assignment 2''

* Implement an linear congruential generator with a = 106, c = 1283, m = 6075 that generates random numbers from 0..1
* Using that and MT: generate 10,000 pairs (dx, dy) with dx, dy each in -0.1 .. +0.1. Generate histograms of dx and dy (say 200 bins). Does it look okay? What would you expect variation to be?
* For 10,000 points: take random walks from (x,y)=(0,0) until exceed radius of 2, then stop. Plot histogram of final angles for the two psuedo random number generators. What do you see?
* Submit makefile, code, plots, git log.

Both assignments due on Thursday Feb 28th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW2===

''Assignment 1''

* Compute numerically (using the GSL):

::∫03 f(x)  dx

:(that is the integral of f(x) from x=0 to x=3)

:with

::f(x) = ln(x) sin(x) e-x

:using three different methods:
# Extended Simpsons' rule
# Gauss-Legendre quadrature
# Monte Carlo sampling

*Hint: what is f(0)?

*Compare the convergence of these methods by increasing number of function evaluations.

*Submit makefile, code, plots, version control log.

''Assignment 2''

* Using an adaptive 4th order Runge-Kutta approach, with a relative accuracy of 1e-4, compute the solution for t = [0,100] of the following set of coupled ODEs (Lorenz oscillator)

::dx/dt = σ(y - x)

::dy/dt = (ρ-z)x-y

::dz/dt = xy - βz

:with σ=10; β=8/3; ρ = 28, and with initial conditions

::x(0) = 10

::y(0) = 20

::z(0) = 30

* Hint: study the GSL documentation.

*Submit makefile, code, plots, version control log.

Both assignments due on Thursday Mar 7th, 2013, at 9:00 am. Email the files to rzon@scinethpc.ca and ljdursi@scinethpc.ca.

===HW3===

Part 1:

The time-explicit formulation of the 1d diffusion equation looks like this:
<math>
\begin{eqnarray*}
q^{n+1} & = & q^n + \frac{D \Delta t}{\Delta x^2}
\left (
\begin{matrix}
-2 & 1 \\
1 & -2 & 1 \\
& 1 & -2 & 1 \\
& & & \cdots & \\
& & & 1 & -2 & 1 \\
& & & & 1 & -2 \\
\end{matrix}
\right ) q^n \\
& = & \left ( 1 + \frac{D \Delta t}{\Delta x^2} A \right ) q^n
\end{eqnarray*}
</math>

what are the eignvalues of the matrix A? What modes would we expect to be amplified/damped by this operator?

* Consider 100 points in the discretization (eg, A is 100x100)
* Calculate the eigenvalues and eigenvectors (using D__EV ; which sort of matrix are we using here?)
* Plot the modes with the largest and smallest absolute-value of eigenvalues, and explain their physical significance
* The numerical method will become unstable with one eigenmode $v$ begins to grow uncontrollably whenever it is present, e.g.
$ \frac{D \Delta t}{\Delta x^2} A v = \frac{D \Delta t}{\Delta x^2} \lambda v > v$. In a timestepping solution, the only way to avoid this for a given physical set of parameters and grid size is to reduce the timestep, $\Delta t$. Use the largest absolute value eigenvalue to place a constraint on $\Delta t$ for stability.

Part 2:

Using the above constraint on $\Delta t$, for a 1d grid of size 100 (eg, a 100x100 matrix A), using lapack, evolve this PDE. Plot and explain results.

* Have an initial condition of $q(x=0,t=0) = 1$, and $q(t=0)$ everywhere else being zero (eg, hot plate just turned on at the left)
* Take ~100 timesteps and plot the the evolution of $q(x,t)$ at 5 times over that period.
* You’ll want to use a matrix multiply to compute the matrix-vector multiply ( http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html). Do multiply in double precision (D__MV). Which should you use?
* The GSL has a cblas interface, http://www.gnu.org/software/gsl/manual/html_node/Level-2-GSL-BLAS-Interface.html ; an example of its use can be found here http://www.gnu.org/software/gsl/manual/html_node/GSL-CBLAS-Examples.html

Important things to know about lapack:
* If you are using an nxn array, the “leading dimension” of the array is n. (This argument is so that you could work on sub-matrices if you wanted)
* Have to make sure the 2d array is contiguous block of memory
* You'll (presumably) want to use the C bindings for LAPACK - [http://www.netlib.org/lapack/lapacke.html lapacke]. Note that the usual C arrays are row-major.

Here's a simple example of calling a LAPACKE routine; note that how the matrix is described (here with a pointer to the data, a leading dimension, and the number of rows and columns) will vary with different types of matrix:

<source lang="cpp">
#include <iostream>
#include <mkl_lapacke.h>

double **matrix(int n,int m);
void free_matrix(double **a);

int main (int argc, const char * argv[])
{

const int n=5; // number of rows, columns of the matrix
const int m = n; // nrows
const int leading_dim_A=n; // leading dimension (# of cols for row major);
// lets us operate on sub-matrices in principle
const int leading_dim_b=n; // similarly for b
double **A;
double *b;

b = new double[leading_dim_b];
A = matrix(n,leading_dim_A);

for (int i=0; i<n; i++)
for (int j=0; j<leading_dim_A; j++)
A[i][j] = 0.;

// let's do a trivial solve
// It should be pretty clear that the solution to this system
// is x = {0,1,2...n-1}

for (int i=0; i<leading_dim_A; i++) {
A[i][i] = 2.;
}

for (int i=0; i<leading_dim_b; i++) {
b[i] = 2*i;
}

const char transpose='N'; //solve Ax=b, not A^T x = b
const int nrhs = 1; // we're only solving 1 right hand side
int info;

// Call DGELS; b will be overwritten with the value of x.
info = LAPACKE_dgels(LAPACK_COL_MAJOR,transpose,m,n,nrhs,
&(A[0][0]),leading_dim_A, &(b[0]),leading_dim_b);

// print results
for(int i=0;i<n;i++)
{
if (i != n/2)
std::cout << " " << b[i] << std::endl;
else
std::cout << "x = " << b[i] << std::endl;
}
return(info);
}

double **matrix(int n,int m) {
double **a = new double * [n];
a[0] = new double [n*m];

for (int i=1; i<n; i++)
a[i] = &a[0][i*m];

return a;
}

void free_matrix(double **a) {
delete[] a[0];
delete[] a;
}
</source>

===HW4===

''Assignment 1''

Trigonometric interpolation uses a n point Fourier series to find values at intermediate points. It is one way of downscaling data, and was a motivation for Gauss, to be applied to planetary motion.

The way it works is:

# You fourier-transform your data
# You add frequecies above the Nyquist frequency (in absolute values), but set all the amplitudes of the new frequencies to zero.
# Note that the frequencies are stored such that eg. fn-1 is a low frequency -1.
# The resulting 2n array can be back transformed, and now gives an interpolated signal.

For this assignment, write an application that will read in an image from a binary file into a 2d double precision array (this will require converting from bytes to doubles), and creates an image twice the size in all directions using trigonometric interpolation. Use a real-to-half-complex version of the fftw (note: in 2d, this version of the fftw mixes fourier components with the same physical magnitude of their wave number k, so this will work).
You can process the red, green and blue values separately.

''Assignment 2''

Write an application which reads an image and performs a low pass filter on the image, i.e., any fourier components with magnitudes k larger than n/8 are to be set to zero, after which the fourier inverse is taken and the image is written out to disk again. Use the same fft technique as in the first assignment.

'''Input image'''

Use [[Media:gauss256.tgz|this image of Gauss]].

'''Image format:'''

Use the following simple PPM format:

First line (ascii): 'P6\n' 
Second line, in ascii, 'width height\n' 
Third line, in ascii, 'maxcolorvalue\n' (this is typically just 255) 
Following that, in binary, are byte-triplets with the red, green and blue values of each pixel. 
Note: in C, the 'unsigned char' data type matches the concept of a byte best (for most machines anyway).

In fact, between the first and second line, one can have comment lines that start with '#'.

=Part 3: High Performance Scientific Computing=

==Prerequisites==

Part 1 or good c++ programming skills, including make and unix/linux prompt experience.

'''Software that you'll need'''

You will need to bring a laptop with a ssh facility. Hands-on parts will be done on SciNet's GPC cluster.

For those who don't have a SciNet account yet, the instructions can be found at http://wiki.scinethpc.ca/wiki/index.php/Essentials#Accounts

==Dates==
March 19, 21, 26, and 28, 2013 
April 2, 4, 9, and 11, 2013

==Topics==
===''Lecture 1:'' Introduction to Parallel Programming ===
:::[[File:Lecture17-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.html]]
:::[[Media:Lecture17-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture17-2013/lecture17-2013.mp4 Video recording]

===''Lecture 2:'' Parallel Computing Paradigms ===

:::[[File:Lecture18-2013-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.html]]
:::[[Media:Lecture18-2013.pdf|Slides]]  /   [http://support.scinet.utoronto.ca/CourseVideo/SCcourse/lecture18-2013/lecture18-2013.mp4 Video recording]  /   [[#HW1_3|homework 1]]

===''Lectures 3,4:'' Shared Memory Programming with OpenMP, part 1,2===

:::[[Media:Lecture19-2013.pdf|Slides]]

===''Lecture 5:'' Distributed Parallel Programming with MPI, part 1===

:::[[Media:Lecture21-2013.pdf|Slides]]

''Lecture 6''   Distributed Parallel Programming with MPI, part 2 
''Lecture 7''   Distributed Parallel Programming with MPI, part 3 
''Lecture 8''   Hybrid OpenMPI+MPI Programming

== Homework assignments ==

=== HW1 ===

* Read the SciNet tutorial (as it pertains to the GPC)
* Read the GPC Quick Start.
* Get the first set of code:
<source lang="bash">
$ cd $SCRATCH
$ git clone /scinet/course/sc3/homework1
$ cd homework1
$ source setup
$ make
</source>
*This contains threaded program 'blurppm' and 266 ppm images to be blurred. Usage:
<source lang="bash">
blurppm INPUTPPM OUTPUTPPM BLURRADIUS NUMBEROFTHREADS
</source>
* Simple test:
<source lang="bash">
$ qsub -l nodes=1:ppn=8,walltime=2:00:00 -I -X -qdebug
$ cd $SCRATCH/homework1
$ time blurppm 001.ppm new001.ppm 30 1
real 0m52.900s
user 0m52.881s
sys 0m0.008s
$ display 001.ppm &
$ display new001.ppm &
</source>
''Assignment 1''
* Time blurppm with a BLURRADIUS ranging from 1 to 41 in steps of 4, and for NUMBEROFTHREADS ranging from 1 to 16. Record the (real) duration of each run.
* Plot the duration as a function of NUMBEROFTHREADS, as well as the speed-up and efficiency.
* Submit script and plots of the duration, speedup and effiency as a function of NUMBEROFTHREADS.
''Assignment 2''
* Use GNU parallel to run blurppm on all 266 images with a radius of 41.
* Investigate different scenarios:
:# Have GNU parallel run 16 at a time with just 1 thread.
:# Have GNU parallel run 8 at a time with 2 threads.
:# Have GNU parallel run 4 at a time with 4 threads.
:# Have GNU parallel run 2 at a time with 8 threads.
:# Have GNU parallel run 1 at a time with 16 threads.
:Record the total time it takes in each of these scenarios.
* Repeat this with a BLURRADIUS of 3.
* Submit scripts, timing data and plots.

=Links=

==Unix==
* Cygwin: http://www.cygwin.com
* Linux Command Line: A Primer (June 2012) [[Media:SS_IntroToShell.pdf|Slides,]] [[Media:SS_IntroToShell.tgz|Files]]
* Intro to unix shell from software carpentry: http://software-carpentry.org/4_0/shell

==C/C++==
* [[One-Day Scientific C++ Class]] at SciNet
* C++ library reference: http://www.cplusplus.com/reference
* C preprocessor: http://www.cprogramming.com/tutorial/cpreprocessor.html
* Boost: http://www.boost.org
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==Git==
* Git: http://git-scm.com
* Version Control: [http://support.scinet.utoronto.ca/CourseVideo/PPPcourse/Thursday_Morning_BP_Revision_Control/Thursday_Morning_BP_Revision_Control.mp4 Video]/ [[Media:Snug_techtalk_revcontrol.pdf | Slides]]
* Git cheat sheet from Git Tower: http://www.git-tower.com/files/cheatsheet/Git_Cheat_Sheet_grey.pdf

==Python==
* Python: http://www.python.org
* IPython: http://ipython.org
* Matplotlib: http://www.matplotlib.org
* Enthought python distribution: http://www.enthought.com/products/edudownload.php 
(this gives you numpy, matplotlib and ipython all installed in one fell swoop)

* Intro to python from software carpentry: http://software-carpentry.org/4_0/python
* Tutorial on matplotlib: http://conference.scipy.org/scipy2011/tutorials.php#jonathan
* Npy file format: https://github.com/numpy/numpy/blob/master/doc/neps/npy-format.txt
* Boost Python tutorial: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/tutorial/doc/html/index.html

==ODEs==
* Integrators for particle based ODEs (i.e. molecular dynamics): http://www.chem.utoronto.ca/~rzon/simcourse/partmd.pdf. '''Focus on 4.1.4 - 4.1.6 for practical aspects.'''
* Numerical algorithm to solve ODEs (General) in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Chapter 16

==Interpolation (2D) ==
* Interpolation in ''Numerical Recipes for C'': http://apps.nrbook.com/c/index.html Pages 123-128
* Wikipedia pages on [http://en.wikipedia.org/wiki/Bilinear_interpolation Bilinear Interpolation] and [http://en.wikipedia.org/wiki/Bicubic_interpolation Bicubic Interpolation] are not bad either.

==BLAS==
* [http://www.tacc.utexas.edu/tacc-projects/gotoblas2 gotoblas]
* [http://math-atlas.sourceforge.net/ ATLAS]

==LAPACK==
* http://www.netlib.org/lapack

==GSL==
* GNU Scientific Library: http://www.gnu.org/s/gsl

==FFT==
* FFTW: http://www.fftw.org

==Top500==
* TOP500 Supercomputing Sites: http://top500.org

==OpenMP==
* OpenMP (open multi-processing) application programming interface for shared memory programming: http://openmp.org

==GNU parallel==
* Official citation: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.
* [[Media:Tech-talk-gnu-parallel.pdf|Slides of the SciNet TechTalk on Gnu Parallel (14 Nov 2012)]]
* The documentation for GNU parallel can be found at http://www.gnu.org/software/parallel/
* Its man page can be found here http://www.gnu.org/software/parallel/man.html
* The man page is also available on the GPC when the gnu-parallel module is loaded, with the command <code>$ man parallel</code>. The man page contains options, such as how to make sure the output is not all scrambled, and examples.

==SciNet==

Anything on this wiki, really, but specifically:
* [[Essentials|SciNet Essentials]]
* [[GPC Quickstart]]
* [[Media:SciNet_Tutorial.pdf |SciNet User Tutorial]]
* [[Software and Libraries]]

==Other Resources==
* [http://galileo.phys.virginia.edu/classes/551.jvn.fall01/goldberg.pdf What Every Computer Scientist Should Know About Floating-Point Arithmetic] - the classic (and extremely comprehensive) overview of the basics of floating point math. The first few pages, in particular, are very useful.
* [http://arxiv.org/abs/1005.4117 Random Numbers In Scientific Computing: An Introduction] by Katzgraber. A very lucid discussion of pseudo random number generators for science.